Investment Securities - Unrealized Loss Position (Details) (USD $) In Thousands, unless otherwise specified	Dec. 31, 2014 position	Dec. 31, 2013 position
Available-for-sale Securities, Continuous Unrealized Loss Position, Fair Value [Abstract]
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value	$ 495,501	$ 684,499
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value	183,763	128,670
Available-for-sale Securities, Continuous Unrealized Loss Position, Aggregate Loss [Abstract]
Available-for-Sale Continuous Unrealized Loss Position Less than 12 Months, Aggregate Losses	(228)	(557)
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Aggregate Losses	(9,983)	(15,430)
Available-for-sale Securities, Continuous Unrealized Loss Position, Qualitative Disclosure [Abstract]
Available-for-sale, Securities in Unrealized Loss Positions, Qualitative Disclosure, Number of Positions	35	64
Investment Securities [Member]
Available-for-sale Securities, Continuous Unrealized Loss Position, Qualitative Disclosure [Abstract]
Available-for-sale securities in a loss position for more than 12 months, qualitative disclosure, percent of amortized cost	95.00%
Available-for-sale, Securities in Unrealized Loss Positions, Qualitative Disclosure, Number of Positions, Greater than or Equal to One Year	15	7
Standard & Poor's, A- Rating [Member]
Available-for-sale Securities, Continuous Unrealized Loss Position, Qualitative Disclosure [Abstract]
Available-for-sale, Securities in Unrealized Loss Positions, Qualitative Disclosure, Number of Positions	1	2
Standard & Poor's, BBB+ Rating [Member]
Available-for-sale Securities, Continuous Unrealized Loss Position, Qualitative Disclosure [Abstract]
Available-for-sale, Securities in Unrealized Loss Positions, Qualitative Disclosure, Number of Positions		1
Floating Interest Rate [Member] | Floating rate auction-rate certificates backed by Government guaranteed student loans
Available-for-sale Securities, Continuous Unrealized Loss Position, Fair Value [Abstract]
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value	0	0
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value	40,576	65,285
Available-for-sale Securities, Continuous Unrealized Loss Position, Aggregate Loss [Abstract]
Available-for-Sale Continuous Unrealized Loss Position Less than 12 Months, Aggregate Losses	0	0
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Aggregate Losses	(6,024)	(8,815)
Floating Interest Rate [Member] | Asset-backed Securities [Member]
Available-for-sale Securities, Continuous Unrealized Loss Position, Fair Value [Abstract]
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value	19,388	50,129
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value	0	0
Available-for-sale Securities, Continuous Unrealized Loss Position, Aggregate Loss [Abstract]
Available-for-Sale Continuous Unrealized Loss Position Less than 12 Months, Aggregate Losses	(37)	(59)
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Aggregate Losses	0	0
Floating Interest Rate [Member] | Corporate Debt Securities [Member]
Available-for-sale Securities, Continuous Unrealized Loss Position, Fair Value [Abstract]
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value		19,982
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value		0
Available-for-sale Securities, Continuous Unrealized Loss Position, Aggregate Loss [Abstract]
Available-for-Sale Continuous Unrealized Loss Position Less than 12 Months, Aggregate Losses		(18)
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Aggregate Losses		0
Floating Interest Rate [Member] | Government/GSE Guaranteed Mortgage Backed Securities [Member]
Available-for-sale Securities, Continuous Unrealized Loss Position, Fair Value [Abstract]
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value	76,100	161,960
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value	76,867	0
Available-for-sale Securities, Continuous Unrealized Loss Position, Aggregate Loss [Abstract]
Available-for-Sale Continuous Unrealized Loss Position Less than 12 Months, Aggregate Losses	(164)	(435)
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Aggregate Losses	(279)	0
Floating Interest Rate [Member] | GSE Subordinated Debt [Member}
Available-for-sale Securities, Continuous Unrealized Loss Position, Fair Value [Abstract]
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value	0	0
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value	66,320	63,385
Available-for-sale Securities, Continuous Unrealized Loss Position, Aggregate Loss [Abstract]
Available-for-Sale Continuous Unrealized Loss Position Less than 12 Months, Aggregate Losses	0	0
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Aggregate Losses	(3,680)	(6,615)
Fixed Interest Rate [Member] | Corporate Debt Securities [Member]
Available-for-sale Securities, Continuous Unrealized Loss Position, Fair Value [Abstract]
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value		10,058
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value		0
Available-for-sale Securities, Continuous Unrealized Loss Position, Aggregate Loss [Abstract]
Available-for-Sale Continuous Unrealized Loss Position Less than 12 Months, Aggregate Losses		(4)
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Aggregate Losses		0
Fixed Interest Rate [Member] | Taxable Municipal Bonds [Member]
Available-for-sale Securities, Continuous Unrealized Loss Position, Fair Value [Abstract]
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value		8,041
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value		0
Available-for-sale Securities, Continuous Unrealized Loss Position, Aggregate Loss [Abstract]
Available-for-Sale Continuous Unrealized Loss Position Less than 12 Months, Aggregate Losses		(3)
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Aggregate Losses		0
Fixed Interest Rate [Member] | Preferred Stock [Member]
Available-for-sale Securities, Continuous Unrealized Loss Position, Fair Value [Abstract]
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value	74,980
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value	0
Available-for-sale Securities, Continuous Unrealized Loss Position, Aggregate Loss [Abstract]
Available-for-Sale Continuous Unrealized Loss Position Less than 12 Months, Aggregate Losses	(11)
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Aggregate Losses	0
Fixed Interest Rate [Member] | USDA Securities [Member]
Available-for-sale Securities, Continuous Unrealized Loss Position, Fair Value [Abstract]
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value		316,273
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value		0
Available-for-sale Securities, Continuous Unrealized Loss Position, Aggregate Loss [Abstract]
Available-for-Sale Continuous Unrealized Loss Position Less than 12 Months, Aggregate Losses		(30)
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Aggregate Losses		0
Fixed Interest Rate [Member] | US Treasury Securities [Member]
Available-for-sale Securities, Continuous Unrealized Loss Position, Fair Value [Abstract]
Available-for-sale Securities, Continuous Unrealized Loss Position, Less than Twelve Months, Fair Value	325,033	118,056
Available-for-sale Securities, Continuous Unrealized Loss Position, Twelve Months or Longer, Fair Value	0	0
Available-for-sale Securities, Continuous Unrealized Loss Position, Aggregate Loss [Abstract]
Available-for-Sale Continuous Unrealized Loss Position Less than 12 Months, Aggregate Losses	(16)	(8)
Available-for-sale Securities, Continuous Unrealized Loss Position, 12 Months or Longer, Aggregate Losses	$ 0	$ 0