MBS and CRT Securities (Details 2)	3 Months Ended
	Mar. 31, 2017 USD ($) Security	Mar. 31, 2016 USD ($)	Dec. 31, 2016 USD ($)
Fair Value
Less than 12 Months	$ 890,140,000
12 Months or more	1,275,967,000
Total	2,166,107,000
Unrealized Losses
Less than 12 Months	8,801,000
12 Months or more	25,702,000
Total	$ 34,503,000
Number of Securities
Less than 12 months (in security) | Security	137
12 months or more (in security) | Security	240
Gross Unrealized Losses	$ 34,503,000		$ 36,398,000
Total OTTI losses	(63,000)	$ 0
OTTI reclassified from OCI	(351,000)	0
Net Impairment Losses Recognized in Earnings	(414,000)	0
Roll-forward of the credit loss component of OTTI
Credit loss component of OTTI at beginning of period	37,305,000	36,820,000
Additions for credit related OTTI not previously recognized	63,000	0
Subsequent additional credit related OTTI recorded	351,000	0
Credit loss component of OTTI at end of period	37,719,000	36,820,000
AOCI from AFS Securities:
Unrealized gain on AFS securities at beginning of period	620,403,000	585,250,000
Reclassification adjustment for MBS sales included in net income	(9,971,000)	(10,485,000)
Reclassification adjustment for OTTI included in net income	(414,000)	0
Change in AOCI from AFS securities	9,084,000	(56,099,000)
Balance at end of period	629,487,000	529,151,000
Total MBS
Fair Value
Less than 12 Months	865,190,000
12 Months or more	1,275,967,000
Total	2,141,157,000
Unrealized Losses
Less than 12 Months	8,723,000
12 Months or more	25,702,000
Total	$ 34,425,000
Number of Securities
Less than 12 months (in security) | Security	136
12 months or more (in security) | Security	240
Gross Unrealized Losses	$ 34,425,000		36,395,000
Agency MBS
Fair Value
Less than 12 Months	654,435,000
12 Months or more	1,099,246,000
Total	1,753,681,000
Unrealized Losses
Less than 12 Months	8,244,000
12 Months or more	23,889,000
Total	$ 32,133,000
Number of Securities
Less than 12 months (in security) | Security	130
12 months or more (in security) | Security	219
Gross Unrealized Losses	$ 32,133,000		31,193,000
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	(8,052,000)	13,226,000
MBS and CRT Securities Interest Income
Coupon interest	26,212,000	32,132,000
Effective yield adjustment	(8,318,000)	(8,135,000)
Interest income	17,894,000	23,997,000
Agency MBS | Fannie Mae
Fair Value
Less than 12 Months	385,791,000
12 Months or more	862,234,000
Total	1,248,025,000
Unrealized Losses
Less than 12 Months	2,985,000
12 Months or more	16,663,000
Total	$ 19,648,000
Number of Securities
Less than 12 months (in security) | Security	82
12 months or more (in security) | Security	154
Gross Unrealized Losses	$ 19,648,000		19,308,000
Agency MBS | Freddie Mac
Fair Value
Less than 12 Months	268,644,000
12 Months or more	237,012,000
Total	505,656,000
Unrealized Losses
Less than 12 Months	5,259,000
12 Months or more	7,226,000
Total	$ 12,485,000
Number of Securities
Less than 12 months (in security) | Security	48
12 months or more (in security) | Security	65
Gross Unrealized Losses	$ 12,485,000		11,885,000
Non-Agency MBS
Fair Value
Less than 12 Months	210,755,000
12 Months or more	176,721,000
Total	387,476,000
Unrealized Losses
Less than 12 Months	479,000
12 Months or more	1,813,000
Total	$ 2,292,000
Number of Securities
Less than 12 months (in security) | Security	6
12 months or more (in security) | Security	21
Gross Unrealized Losses	$ 2,292,000		5,202,000
Credit-related OTTI losses		0
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	27,521,000	(58,840,000)
Sales of MBS
Proceeds from sale of MBS categorized as available-for-sale	21,600,000	32,000,000
Gains on sales of MBS	10,000,000	9,700,000
Non-Agency MBS | Discount Designated as Credit Reserve and OTTI
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period	(694,241,000)	(787,541,000)
Accretion of discount	0	0
Realized credit losses	12,324,000	18,050,000
Purchases	0	(4,294,000)
Sales	19,741,000	12,020,000
Net impairment losses recognized in earnings	(414,000)	0
Transfers/release of credit reserve	9,253,000	4,201,000
Balance at end of period	(653,337,000)	(757,564,000)
Non-Agency MBS | Accretable Discount
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period	(278,191,000)	(312,182,000)
Accretion of discount	21,616,000	21,406,000
Realized credit losses	0	0
Purchases	0	1,606,000
Sales	(3,897,000)	12,040,000
Net impairment losses recognized in earnings	0	0
Transfers/release of credit reserve	(9,253,000)	(4,201,000)
Balance at end of period	(269,725,000)	(281,331,000)
Non-Agency MBS | Expected to Recover Par
Fair Value
Less than 12 Months	207,251,000
12 Months or more	126,702,000
Total	333,953,000
Unrealized Losses
Less than 12 Months	399,000
12 Months or more	585,000
Total	$ 984,000
Number of Securities
Less than 12 months (in security) | Security	4
12 months or more (in security) | Security	12
Gross Unrealized Losses	$ 984,000		2,368,000
Non-Agency MBS | Expected to Recover Less Than Par
Fair Value
Less than 12 Months	3,504,000
12 Months or more	50,019,000
Total	53,523,000
Unrealized Losses
Less than 12 Months	80,000
12 Months or more	1,228,000
Total	$ 1,308,000
Number of Securities
Less than 12 months (in security) | Security	2
12 months or more (in security) | Security	9
Gross Unrealized Losses	$ 1,308,000		2,834,000
Legacy Non-Agency MBS
MBS and CRT Securities Interest Income
Coupon interest	34,662,000	40,309,000
Effective yield adjustment	21,442,000	19,913,000
Interest income	56,104,000	60,222,000
3 Year Step-up securities
MBS and CRT Securities Interest Income
Coupon interest	25,965,000	24,370,000
Effective yield adjustment	174,000	1,560,000
Interest income	26,139,000	25,930,000
CRT securities
Fair Value
Less than 12 Months	24,950,000
12 Months or more	0
Total	24,950,000
Unrealized Losses
Less than 12 Months	78,000
12 Months or more	0
Total	$ 78,000
Number of Securities
Less than 12 months (in security) | Security	1
12 months or more (in security) | Security	0
Gross Unrealized Losses	$ 78,000		$ 3,000
MBS and CRT Securities Interest Income
Coupon interest	5,257,000	2,356,000
Effective yield adjustment	1,119,000	336,000
Interest income	6,376,000	$ 2,692,000
CRT securities | Fair value option elected, debt securities
Fair Value
Total	25,000,000
Number of Securities
Gross Unrealized Losses	$ 78,000