Segment Information - Schedule of Operating Results by Segment (Details) (USD $) In Thousands, unless otherwise specified	12 Months Ended
	Dec. 31, 2013	Dec. 31, 2012	Dec. 31, 2011
Segment Reporting Information [Line Items]
Net interest income (loss)	$ 221,025	$ 235,315	$ 273,234
Provision for loan losses	3,415	35,419	63,272
Other income	73,277	92,403	94,910
Other expense	188,529	187,968	188,317
Income (loss) before taxes	102,358	104,331	116,555
Income before income taxes	102,358	104,331	116,555
Income taxes (benefit)	25,131	25,701	34,415
Net income	77,227	78,630	82,140
Assets	6,638,347	6,642,803	6,972,245
Assets held for sale (1)			382,462	[1]
Loans	4,620,505	4,450,322	4,317,099
Total loans	4,620,505	4,450,322	4,317,099
Deposits	4,789,994	4,716,032	4,465,114
Liabilities held for sale (2)			536,186	[2]
PNB
Segment Reporting Information [Line Items]
Net interest income (loss)	210,781	221,758	236,282
Provision for loan losses	14,039	16,678	30,220
Other income	70,841	70,739	90,982
Other expense	165,665	156,516	146,235
Income (loss) before taxes	101,918	119,303	150,809
Income before income taxes	101,918	119,303	150,809
Income taxes (benefit)	26,324	32,197	43,958
Net income	75,594	87,106	106,851
Assets	6,524,098	6,502,579	6,281,747
Assets held for sale (1)			0	[1]
Loans	4,559,406	4,369,173	4,172,424
Total loans	4,559,406	4,369,173	4,172,424
Deposits	4,896,405	4,814,107	4,611,646
Liabilities held for sale (2)			0	[2]
VB
Segment Reporting Information [Line Items]
Net interest income (loss)	0	0	27,078
Provision for loan losses	0	0	31,052
Other income	0	0	6,617
Other expense	0	0	31,379
Income (loss) before taxes	0	0	(28,736)
Income before income taxes	0	0	(28,736)
Income taxes (benefit)	0	0	(6,210)
Net income	0	0	(22,526)
Assets	0	0	650,935
Assets held for sale (1)			382,462	[1]
Loans	0	0	123,883
Total loans	0	0	123,883
Deposits	0	0	32
Liabilities held for sale (2)			536,186	[2]
GFSC
Segment Reporting Information [Line Items]
Net interest income (loss)	8,741	9,156	8,693
Provision for loan losses	1,175	859	2,000
Other income	11	0	0
Other expense	3,133	2,835	2,506
Income (loss) before taxes	4,444	5,462	4,187
Income before income taxes	4,444	5,462	4,187
Income taxes (benefit)	1,556	1,912	1,466
Net income	2,888	3,550	2,721
Assets	47,115	49,926	46,682
Assets held for sale (1)			0	[1]
Loans	47,228	50,082	47,111
Total loans	47,228	50,082	47,111
Deposits	7,159	8,358	8,013
Liabilities held for sale (2)			0	[2]
SEPH
Segment Reporting Information [Line Items]
Net interest income (loss)	(1,325)	(341)	(974)
Provision for loan losses	(11,799)	17,882	0
Other income	1,956	21,431	(3,039)
Other expense	12,211	22,032	1,082
Income (loss) before taxes	219	(18,824)	(5,095)
Income before income taxes	219	(18,824)	(5,095)
Income taxes (benefit)	77	(6,603)	(1,784)
Net income	142	(12,221)	(3,311)
Assets	72,781	104,428	34,989
Assets held for sale (1)			0	[1]
Loans	38,014	59,178	0
Total loans	38,014	59,178	0
Deposits	0	0	0
Liabilities held for sale (2)			0	[2]
All Other
Segment Reporting Information [Line Items]
Net interest income (loss)	2,828	4,742	2,155
Provision for loan losses	0	0	0
Other income	469	233	350
Other expense	7,520	6,585	7,115
Income (loss) before taxes	(4,223)	(1,610)	(4,610)
Income before income taxes	(4,223)	(1,610)	(4,610)
Income taxes (benefit)	(2,826)	(1,805)	(3,015)
Net income	(1,397)	195	(1,595)
Assets	(5,647)	(14,130)	(42,108)
Assets held for sale (1)			0	[1]
Loans	(24,143)	(28,111)	(26,319)
Total loans	(24,143)	(28,111)	(26,319)
Deposits	(113,570)	(106,433)	(154,577)
Liabilities held for sale (2)			$ 0	[2]

[1] The assets held for sale represent the loans and other assets at Vision that were sold in the first quarter of 2012.

[2] The liabilities held for sale represent the deposits and other liabilities at Vision that were sold in the first quarter of 2012.