Investment Securities (Details)	3 Months Ended		6 Months Ended
	Jun. 30, 2016 USD ($) fund item	Jun. 30, 2015 USD ($)	Jun. 30, 2016 USD ($) item	Jun. 30, 2015 USD ($)	Dec. 31, 2015 USD ($) item
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	$ 199,948,000		$ 199,948,000		$ 224,108,000
Gross unrealized gains	4,449,000		4,449,000		2,205,000
Gross unrealized losses	(339,000)		(339,000)		(2,246,000)
Available-for-sale Securities, Total	204,058,000		204,058,000		224,067,000
Amortized cost [Abstract]
Within 1 year	1,003,000		1,003,000
After 1 but within 5 years	10,811,000		10,811,000
After 5 but within 10 years	9,091,000		9,091,000
After 10 years	4,113,000		4,113,000
Total available for sale debt securities	198,045,000		198,045,000
Available-for-sale Equity Securities, Amortized Cost Basis	1,903,000		1,903,000
Total available for sale securities	199,948,000		199,948,000		224,108,000
Fair value [Abstract]
Within 1 year	1,001,000		1,001,000
After 1 but within 5 years	10,827,000		10,827,000
After 5 but within 10 years	9,319,000		9,319,000
After 10 years	4,220,000		4,220,000
Total available for sale debt securities	202,155,000		202,155,000
No contractual maturity	1,903,000		1,903,000
Fair value	204,058,000		204,058,000		224,067,000
Proceeds from sales of AFS investment securities	683,000	$ 5,800,000	10,705,000	$ 24,506,000
Gross realized gains and losses on the sale of securities recognized in earnings [Abstract]
Gross realized gains	779,000	385,000	1,146,000	572,000
Gross realized losses	(44,000)	0	(44,000)	0
Net realized gains	735,000	385,000	1,102,000	572,000
Other than Temporary Impairment Losses, Investments, Portion Recognized in Earnings, Net, Available-for-sale Securities	146,000	$ 0	146,000	0
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value	11,835,000		11,835,000		106,022,000
12 months or longer, Fair value	12,033,000		12,033,000		37,100,000
Total, Fair value	23,868,000		23,868,000		143,122,000
Less than 12 months, Gross unrealized losses	(126,000)		(126,000)		(1,134,000)
12 months or longer, Gross unrealized losses	(213,000)		(213,000)		(1,112,000)
Total, Gross unrealized losses	$ (339,000)		$ (339,000)		$ (2,246,000)
Less than 12 months, number of positions | item	6		6		43
12 months or longer, number of positions | item	5		5		12
Total , number of positions | item	11		11		55
Fair value	$ 204,058,000		$ 204,058,000		$ 224,067,000
Number of securities in an unrealized loss position for less than twelve months | item	6		6		43
Number of securities in an unrealized loss position for more than twelve months | item	5		5		12
Credit related impairment losses on debt securities held for which a portion of OTTI was recognized in other comprehensive income [Roll Forward]
Reductions for securities sold during the period (realized)			$ 0	$ 0
U.S. Government Agencies [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	$ 13,594,000		13,594,000		$ 26,127,000
Gross unrealized gains	33,000		33,000		0
Gross unrealized losses	0		0		(564,000)
Available-for-sale Securities, Total	13,627,000		13,627,000		25,563,000
Amortized cost [Abstract]
Total available for sale securities	13,594,000		13,594,000		26,127,000
Fair value [Abstract]
Fair value	13,627,000		13,627,000		25,563,000
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value					6,681,000
12 months or longer, Fair value					18,882,000
Total, Fair value					25,563,000
Less than 12 months, Gross unrealized losses					(57,000)
12 months or longer, Gross unrealized losses					(507,000)
Total, Gross unrealized losses					$ (564,000)
Less than 12 months, number of positions | item					2
12 months or longer, number of positions | item					6
Total , number of positions | item					8
Fair value	13,627,000		13,627,000		$ 25,563,000
Number of securities in an unrealized loss position for less than twelve months | item					2
Number of securities in an unrealized loss position for more than twelve months | item					6
Mortgage-backed Securities-residential [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	10,145,000		10,145,000		$ 11,002,000
Gross unrealized gains	357,000		357,000		106,000
Gross unrealized losses	0		0		(50,000)
Available-for-sale Securities, Total	10,502,000		10,502,000		11,058,000
Amortized cost [Abstract]
Available for sale securities Debt maturities, amortized cost basis	10,145,000		10,145,000
Total available for sale securities	10,145,000		10,145,000		11,002,000
Fair value [Abstract]
Available-for-sale Securities, Debt Maturities, without Single Maturity Date, Fair Value	10,502,000		10,502,000
Fair value	10,502,000		10,502,000		11,058,000
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value					5,140,000
12 months or longer, Fair value					2,574,000
Total, Fair value					7,714,000
Less than 12 months, Gross unrealized losses					(5,000)
12 months or longer, Gross unrealized losses					(45,000)
Total, Gross unrealized losses					$ (50,000)
Less than 12 months, number of positions | item					2
12 months or longer, number of positions | item					1
Total , number of positions | item					3
Fair value	10,502,000		10,502,000		$ 11,058,000
Number of securities in an unrealized loss position for less than twelve months | item					2
Number of securities in an unrealized loss position for more than twelve months | item					1
Collateralized Mortgage Obligations, Issued or Guaranteed by U.S. Government Agencies [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	160,639,000		160,639,000		$ 170,764,000
Gross unrealized gains	3,639,000		3,639,000		1,524,000
Gross unrealized losses	(286,000)		(286,000)		(1,554,000)
Available-for-sale Securities, Total	163,992,000		163,992,000		170,734,000
Amortized cost [Abstract]
Available for sale securities Debt maturities, amortized cost basis	160,639,000		160,639,000
Total available for sale securities	160,639,000		160,639,000		170,764,000
Fair value [Abstract]
Available-for-sale Securities, Debt Maturities, without Single Maturity Date, Fair Value	163,992,000		163,992,000
Fair value	163,992,000		163,992,000		170,734,000
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value	8,801,000		8,801,000		87,020,000
12 months or longer, Fair value	10,705,000		10,705,000		15,644,000
Total, Fair value	19,506,000		19,506,000		102,664,000
Less than 12 months, Gross unrealized losses	(77,000)		(77,000)		(994,000)
12 months or longer, Gross unrealized losses	(209,000)		(209,000)		(560,000)
Total, Gross unrealized losses	$ (286,000)		$ (286,000)		$ (1,554,000)
Less than 12 months, number of positions | item	3		3		31
12 months or longer, number of positions | item	3		3		5
Total , number of positions | item	6		6		36
Fair value	$ 163,992,000		$ 163,992,000		$ 170,734,000
Number of securities in an unrealized loss position for less than twelve months | item	3		3		31
Number of securities in an unrealized loss position for more than twelve months | item	3		3		5
Mortgage-backed Securities, Issued by Private Enterprises [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	$ 2,243,000		$ 2,243,000		$ 2,729,000
Gross unrealized gains	51,000		51,000		1,000
Gross unrealized losses	0		0		(26,000)
Available-for-sale Securities, Total	2,294,000		2,294,000		2,704,000
Amortized cost [Abstract]
Available for sale securities Debt maturities, amortized cost basis	2,243,000		2,243,000
Total available for sale securities	2,243,000		2,243,000		2,729,000
Fair value [Abstract]
Available-for-sale Securities, Debt Maturities, without Single Maturity Date, Fair Value	2,294,000		2,294,000
Fair value	2,294,000		2,294,000		2,704,000
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value					1,101,000
12 months or longer, Fair value					0
Total, Fair value					1,101,000
Less than 12 months, Gross unrealized losses					(26,000)
12 months or longer, Gross unrealized losses					0
Total, Gross unrealized losses					$ (26,000)
Less than 12 months, number of positions | item					1
12 months or longer, number of positions | item					0
Total , number of positions | item					1
Fair value	2,294,000		2,294,000		$ 2,704,000
Number of securities in an unrealized loss position for less than twelve months | item					1
Number of securities in an unrealized loss position for more than twelve months | item					0
Corporate Bonds [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	1,700,000		1,700,000		$ 1,500,000
Gross unrealized gains	94,000		94,000		56,000
Gross unrealized losses	0		0		0
Available-for-sale Securities, Total	1,794,000		1,794,000		1,556,000
Amortized cost [Abstract]
Total available for sale securities	1,700,000		1,700,000		1,500,000
Fair value [Abstract]
Fair value	1,794,000		1,794,000		1,556,000
Available for sale securities, continuous unrealized loss position [Abstract]
Fair value	1,794,000		1,794,000		1,556,000
Municipal Bonds [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	9,724,000		9,724,000		9,910,000
Gross unrealized gains	275,000		275,000		73,000
Gross unrealized losses	(53,000)		(53,000)		(52,000)
Available-for-sale Securities, Total	9,946,000		9,946,000		9,931,000
Amortized cost [Abstract]
Total available for sale securities	9,724,000		9,724,000		9,910,000
Fair value [Abstract]
Fair value	9,946,000		9,946,000		9,931,000
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value	3,034,000		3,034,000		6,080,000
12 months or longer, Fair value	1,328,000		1,328,000		0
Total, Fair value	4,362,000		4,362,000		6,080,000
Less than 12 months, Gross unrealized losses	(49,000)		(49,000)		(52,000)
12 months or longer, Gross unrealized losses	(4,000)		(4,000)		0
Total, Gross unrealized losses	$ (53,000)		$ (53,000)		$ (52,000)
Less than 12 months, number of positions | item	3		3		7
12 months or longer, number of positions | item	2		2		0
Total , number of positions | item	5		5		7
Fair value	$ 9,946,000		$ 9,946,000		$ 9,931,000
Number of securities in an unrealized loss position for less than twelve months | item	3		3		7
Number of securities in an unrealized loss position for more than twelve months | item	2		2		0
Private Equity Funds [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	$ 1,877,000		$ 1,877,000		$ 2,050,000
Gross unrealized gains	0		0		445,000
Gross unrealized losses	0		0		0
Available-for-sale Securities, Total	1,877,000		1,877,000		2,495,000
Amortized cost [Abstract]
Total available for sale securities	1,877,000		1,877,000		2,050,000
Fair value [Abstract]
Fair value	1,877,000		1,877,000		2,495,000
Available for sale securities, continuous unrealized loss position [Abstract]
Fair value	1,877,000		1,877,000		2,495,000
Impairment charges on private equity funds	$ 146,000
Number of private equity funds that had impairment charges recorded on | fund	2
Common Stock [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	$ 26,000		26,000		26,000
Gross unrealized gains	0		0		0
Gross unrealized losses	0		0		0
Available-for-sale Securities, Total	26,000		26,000		26,000
Amortized cost [Abstract]
Total available for sale securities	26,000		26,000		26,000
Fair value [Abstract]
Fair value	26,000		26,000		26,000
Available for sale securities, continuous unrealized loss position [Abstract]
Fair value	$ 26,000		$ 26,000		$ 26,000