Investment Securities - Contractual Maturity - (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015
Amortized cost [Abstract]
After 1 but within 5 years	$ 5,282
After 5 but within 10 years	5,097
After 10 years	1,000
Total available for sale debt securities	169,151
Available-for-sale Equity Securities, Amortized Cost Basis	1,847
Total available for sale securities	170,998	$ 224,108
Fair value [Abstract]
After 1 but within 5 years	5,233
After 5 but within 10 years	5,141
After 10 years	1,001
Total available for sale debt securities	168,007
No contractual maturity	1,847
Fair value	169,854	224,067
Proceeds from sales of AFS investment securities	36,436	47,018
U.S. Government Agencies [Member]
Amortized cost [Abstract]
Total available for sale securities	988	26,127
Fair value [Abstract]
Fair value	966	25,563
Mortgage-backed Securities-residential [Member]
Amortized cost [Abstract]
Available for sale securities Debt maturities, amortized cost basis	6,985
Total available for sale securities	6,985	11,002
Fair value [Abstract]
Available-for-sale Securities, Debt Maturities, without Single Maturity Date, Fair Value	7,038
Fair value	7,038	11,058
Collateralized Mortgage Obligations, Issued or Guaranteed by U.S. Government Agencies [Member]
Amortized cost [Abstract]
Available for sale securities Debt maturities, amortized cost basis	146,666
Total available for sale securities	146,666	170,764
Fair value [Abstract]
Available-for-sale Securities, Debt Maturities, without Single Maturity Date, Fair Value	145,559
Fair value	145,559	170,734
Mortgage-backed Securities, Issued by Private Enterprises [Member]
Amortized cost [Abstract]
Total available for sale debt securities	4,121
Total available for sale securities	4,121	2,729
Fair value [Abstract]
Available-for-sale Securities, Debt Maturities, without Single Maturity Date, Fair Value	4,035
Fair value	4,035	2,704
Corporate Bonds [Member]
Amortized cost [Abstract]
Total available for sale securities	1,700	1,500
Fair value [Abstract]
Fair value	1,701	1,556
Municipal Bonds [Member]
Amortized cost [Abstract]
Total available for sale securities	8,691	9,910
Fair value [Abstract]
Fair value	8,708	9,931
Private Equity Funds [Member]
Amortized cost [Abstract]
Total available for sale securities	1,821	2,050
Fair value [Abstract]
Fair value	1,821	2,495
Common Stock [Member]
Amortized cost [Abstract]
Total available for sale securities	26	26
Fair value [Abstract]
Fair value	$ 26	$ 26