Loans (Details) (USD $) In Thousands, unless otherwise specified	3 Months Ended		6 Months Ended
	Jun. 30, 2013	Jun. 30, 2012	Jun. 30, 2013	Jun. 30, 2012	Dec. 31, 2012
Allowance for loan losses:
Individually evaluated for impairment	$ 10,607		$ 10,607		$ 12,506
Collectively evaluated for probable incurred losses	26,597		26,597		44,110
Total allowance for loan losses	37,204		37,204		56,616
Portfolio loans:
Individually evaluated for impairment	142,864		142,864		173,233
Collectively evaluated for probable incurred losses	762,864		762,864		936,435
Total Portfolio Loans	905,728		905,728		1,109,668
Summarize activity in the allowance for loan losses [Roll Forward]
Beginning balance	52,865	71,009	56,616	76,344
Less allowance for deconsolidated subsidiary-Note H	0		(2,157)
Charge-offs	(5,582)	(12,197)	(12,426)	(25,824)
Recoveries	2,117	5,423	7,151	13,054
Net charge-offs	(3,465)	(6,774)	(5,275)	(12,770)
Provision for loan losses	(12,196)	230	(11,980)	891
Ending balance	37,204	64,465	37,204	64,465
Secured by Real Estate [Member]
Portfolio loans:
Total Portfolio Loans	810,796		810,796		981,117
Commercial [Member] | Secured by Real Estate [Member]
Allowance for loan losses:
Individually evaluated for impairment	5,882		5,882		7,444
Collectively evaluated for probable incurred losses	8,645		8,645		10,594
Total allowance for loan losses	14,527		14,527		18,038
Portfolio loans:
Individually evaluated for impairment	98,529		98,529		114,039
Collectively evaluated for probable incurred losses	491,344		491,344		590,090
Total Portfolio Loans	589,873		589,873		704,129
Summarize activity in the allowance for loan losses [Roll Forward]
Beginning balance	18,260	26,198	18,038	32,854
Less allowance for deconsolidated subsidiary-Note H	0		(765)
Charge-offs	(3,362)	(3,478)	(5,701)	(8,918)
Recoveries	694	1,569	2,260	3,959
Net charge-offs	(2,668)	(1,909)	(3,441)	(4,959)
Provision for loan losses	559	2,164	1,141	2,952
Net change in unallocated allowance	(1,624)	(940)	(446)	(5,334)
Ending balance	14,527	25,513	14,527	25,513
Residential (including multi-family) [Member] | Secured by Real Estate [Member]
Allowance for loan losses:
Individually evaluated for impairment	2,530		2,530		2,668
Collectively evaluated for probable incurred losses	4,845		4,845		6,796
Total allowance for loan losses	7,375		7,375		9,464
Portfolio loans:
Individually evaluated for impairment	24,271		24,271		34,763
Collectively evaluated for probable incurred losses	158,355		158,355		195,092
Total Portfolio Loans	182,626		182,626		229,855
Summarize activity in the allowance for loan losses [Roll Forward]
Beginning balance	7,994	14,462	9,464	17,415
Less allowance for deconsolidated subsidiary-Note H	0		(869)
Charge-offs	(1,280)	(2,369)	(2,561)	(6,476)
Recoveries	580	538	1,666	3,627
Net charge-offs	(700)	(1,831)	(895)	(2,849)
Provision for loan losses	582	(249)	(188)	(714)
Net change in unallocated allowance	(501)	(315)	(137)	(1,785)
Ending balance	7,375	12,067	7,375	12,067
Construction, Land Development and Other Land [Member] | Secured by Real Estate [Member]
Allowance for loan losses:
Individually evaluated for impairment	1,121		1,121		1,011
Collectively evaluated for probable incurred losses	932		932		1,310
Total allowance for loan losses	2,053		2,053		2,321
Portfolio loans:
Individually evaluated for impairment	9,630		9,630		11,765
Collectively evaluated for probable incurred losses	28,667		28,667		35,368
Total Portfolio Loans	38,297		38,297		47,133
Summarize activity in the allowance for loan losses [Roll Forward]
Beginning balance	2,238	7,690	2,321	8,727
Less allowance for deconsolidated subsidiary-Note H	0		(53)
Charge-offs	(435)	(1,190)	(2,715)	(3,340)
Recoveries	153	362	1,936	1,125
Net charge-offs	(282)	(828)	(779)	(2,215)
Provision for loan losses	202	(3,039)	593	(2,286)
Net change in unallocated allowance	(105)	(86)	(29)	(489)
Ending balance	2,053	3,737	2,053	3,737
Commercial And Other Business-Purpose Loans [Member]
Allowance for loan losses:
Individually evaluated for impairment	1,014		1,014		1,327
Collectively evaluated for probable incurred losses	2,042		2,042		3,563
Total allowance for loan losses	3,056		3,056		4,890
Portfolio loans:
Individually evaluated for impairment	10,353		10,353		12,577
Collectively evaluated for probable incurred losses	74,547		74,547		105,997
Total Portfolio Loans	84,900		84,900		118,574
Summarize activity in the allowance for loan losses [Roll Forward]
Beginning balance	4,882	10,666	4,890	12,283
Less allowance for deconsolidated subsidiary-Note H	0		(422)
Charge-offs	(220)	(4,470)	(1,031)	(6,183)
Recoveries	668	2,876	1,213	4,199
Net charge-offs	448	(1,594)	182	(1,984)
Provision for loan losses	(2,042)	490	(1,530)	69
Net change in unallocated allowance	(232)	(172)	(64)	(978)
Ending balance	3,056	9,390	3,056	9,390
Consumer [Member]
Allowance for loan losses:
Individually evaluated for impairment	60		60		56
Collectively evaluated for probable incurred losses	365		365		948
Total allowance for loan losses	425		425		1,004
Portfolio loans:
Individually evaluated for impairment	81		81		89
Collectively evaluated for probable incurred losses	6,023		6,023		7,847
Total Portfolio Loans	6,104		6,104		7,936
Summarize activity in the allowance for loan losses [Roll Forward]
Beginning balance	259	394	1,004	601
Less allowance for deconsolidated subsidiary-Note H	0		(48)
Charge-offs	(285)	(11)	(411)	(228)
Recoveries	22	78	76	137
Net charge-offs	(263)	67	(335)	(91)
Provision for loan losses	446	(175)	(191)	(169)
Net change in unallocated allowance	(17)	(12)	(5)	(67)
Ending balance	425	274	425	274
Other [Member]
Allowance for loan losses:
Collectively evaluated for probable incurred losses	390		390		4
Total allowance for loan losses	390		390		4
Portfolio loans:
Collectively evaluated for probable incurred losses	3,928		3,928		2,041
Total Portfolio Loans	3,928		3,928		2,041
Summarize activity in the allowance for loan losses [Roll Forward]
Beginning balance	141	13	4	17
Charge-offs	0	(679)	(7)	(679)
Recoveries	0	0	0	7
Net charge-offs	0	(679)	(7)	(672)
Provision for loan losses	257	1,039	395	1,039
Net change in unallocated allowance	(8)	(3)	(2)	(14)
Ending balance	390	370	390	370
Unallocated [Member]
Allowance for loan losses:
Collectively evaluated for probable incurred losses	9,378		9,378		20,895
Total allowance for loan losses	9,378		9,378		20,895
Summarize activity in the allowance for loan losses [Roll Forward]
Beginning balance	19,091	11,586	20,895	4,447
Provision for loan losses	(12,200)		(12,200)
Net change in unallocated allowance	2,487	1,528	683	8,667
Ending balance	$ 9,378	$ 13,114	$ 9,378	$ 13,114