Fair Value - Financial Assets and Liabilities Measured on a Recurring Basis (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Dec. 31, 2015
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale	$ 369,999	$ 385,079
Agency mortgage-backed securities
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale	352,552	324,230
U.S. Treasury
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale		30,003
Trust preferred securities
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale	16,425	15,132
U.S. Government sponsored entities
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale		9,041
Corporate bonds
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale	1,022	6,673
Recurring basis
Financial Assets and Liabilities Measured on a Recurring Basis
I/O strip receivables	1,101	1,367
Transfers between Level 1 and Level 2
Transfers between Level 1 and Level 2	0	0
Transfers between Level 2 and Level 1	0	0
Recurring basis | Agency mortgage-backed securities
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale	352,552	324,230
Recurring basis | U.S. Treasury
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale		30,003
Recurring basis | Trust preferred securities
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale	16,425	15,132
Recurring basis | U.S. Government sponsored entities
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale		9,041
Recurring basis | Corporate bonds
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale	1,022	6,673
Recurring basis | Quoted Prices in Active Markets for Identical Assets (Level 1) | U.S. Treasury
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale		30,003
Recurring basis | Significant Other Observable Inputs (Level 2)
Financial Assets and Liabilities Measured on a Recurring Basis
I/O strip receivables	1,101	1,367
Recurring basis | Significant Other Observable Inputs (Level 2) | Agency mortgage-backed securities
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale	352,552	324,230
Recurring basis | Significant Other Observable Inputs (Level 2) | Trust preferred securities
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale	16,425	15,132
Recurring basis | Significant Other Observable Inputs (Level 2) | U.S. Government sponsored entities
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale		9,041
Recurring basis | Significant Other Observable Inputs (Level 2) | Corporate bonds
Financial Assets and Liabilities Measured on a Recurring Basis
Securities available-for-sale	$ 1,022	$ 6,673