MBS and CRT Securities (Details 2)		3 Months Ended		6 Months Ended
	Dec. 31, 2014 USD ($)	Jun. 30, 2016 USD ($) Security	Jun. 30, 2015 USD ($)	Jun. 30, 2016 USD ($) Security	Jun. 30, 2015 USD ($)	Dec. 31, 2015 USD ($)
Fair Value
Less than 12 Months		$ 1,227,176,000		$ 1,227,176,000
12 Months or more		2,109,858,000		2,109,858,000
Total		3,337,034,000		3,337,034,000
Unrealized Losses
Less than 12 Months		7,118,000		7,118,000
12 Months or more		18,665,000		18,665,000
Total		$ 25,783,000		$ 25,783,000
Number of Securities
Less than 12 months (in security) | Security		128		128
12 months or more (in security) | Security		240		240
Gross Unrealized Losses		$ 25,783,000		$ 25,783,000		$ 71,929,000
Total OTTI losses		0	$ (130,000)	0	$ (525,000)
OTTI reclassified from OCI		0	(168,000)	0	(180,000)
Net Impairment Losses Recognized in Earnings		0	(298,000)	0	(705,000)
Roll-forward of the credit loss component of OTTI
Credit loss component of OTTI at beginning of period		36,820,000		36,820,000
Additions for credit related OTTI not previously recognized		0		0
Subsequent additional credit related OTTI recorded		0		0
Credit loss component of OTTI at end of period		36,820,000		36,820,000
Changes in the components of the purchase discount on Non-Agency MBS
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing	$ 4,500,000	0	0	0	4,537,000
Proceeds from settlement		61,800,000
AOCI from AFS Securities:
Unrealized gain on AFS securities at beginning of period		529,151,000	850,257,000	585,250,000	813,515,000
Reclassification adjustment for MBS sales included in net income		(8,688,000)	(7,863,000)	(19,651,000)	(14,429,000)
Reclassification adjustment for OTTI included in net income		0	(298,000)	0	(705,000)
Change in AOCI from AFS securities		96,546,000	(91,106,000)	40,447,000	(54,364,000)
Balance at end of period	813,515,000	625,697,000	759,151,000	625,697,000	759,151,000
Total MBS
Fair Value
Less than 12 Months		1,175,618,000		1,175,618,000
12 Months or more		2,105,150,000		2,105,150,000
Total		3,280,768,000		3,280,768,000
Unrealized Losses
Less than 12 Months		6,822,000		6,822,000
12 Months or more		18,372,000		18,372,000
Total		$ 25,194,000		$ 25,194,000
Number of Securities
Less than 12 months (in security) | Security		114		114
12 months or more (in security) | Security		239		239
Gross Unrealized Losses		$ 25,194,000		$ 25,194,000		68,782,000
Agency MBS
Fair Value
Less than 12 Months		443,802,000		443,802,000
12 Months or more		1,263,915,000		1,263,915,000
Total		1,707,717,000		1,707,717,000
Unrealized Losses
Less than 12 Months		3,352,000		3,352,000
12 Months or more		8,241,000		8,241,000
Total		$ 11,593,000		$ 11,593,000
Number of Securities
Less than 12 months (in security) | Security		75		75
12 months or more (in security) | Security		196		196
Gross Unrealized Losses		$ 11,593,000		$ 11,593,000		40,425,000
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net		11,572,000	(25,250,000)	24,798,000	(12,365,000)
MBS and CRT Securities Interest Income
Coupon interest		30,848,000	37,646,000	62,980,000	78,388,000
Effective yield adjustment		(9,256,000)	(11,907,000)	(17,391,000)	(20,976,000)
Interest income		21,592,000	25,739,000	45,589,000	57,412,000
Agency MBS | Fannie Mae
Fair Value
Less than 12 Months		443,802,000		443,802,000
12 Months or more		890,245,000		890,245,000
Total		1,334,047,000		1,334,047,000
Unrealized Losses
Less than 12 Months		3,352,000		3,352,000
12 Months or more		6,135,000		6,135,000
Total		$ 9,487,000		$ 9,487,000
Number of Securities
Less than 12 months (in security) | Security		75		75
12 months or more (in security) | Security		125		125
Gross Unrealized Losses		$ 9,487,000		$ 9,487,000		25,830,000
Agency MBS | Freddie Mac
Fair Value
Less than 12 Months		0		0
12 Months or more		373,670,000		373,670,000
Total		373,670,000		373,670,000
Unrealized Losses
Less than 12 Months		0		0
12 Months or more		2,106,000		2,106,000
Total		$ 2,106,000		$ 2,106,000
Number of Securities
Less than 12 months (in security) | Security		0		0
12 months or more (in security) | Security		71		71
Gross Unrealized Losses		$ 2,106,000		$ 2,106,000		14,595,000
Non-Agency MBS
Fair Value
Less than 12 Months		731,816,000		731,816,000
12 Months or more		841,235,000		841,235,000
Total		1,573,051,000		1,573,051,000
Unrealized Losses
Less than 12 Months		3,470,000		3,470,000
12 Months or more		10,131,000		10,131,000
Total		$ 13,601,000		$ 13,601,000
Number of Securities
Less than 12 months (in security) | Security		39		39
12 months or more (in security) | Security		43		43
Gross Unrealized Losses		$ 13,601,000		$ 13,601,000		28,357,000
Credit-related OTTI losses		0		0
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net		93,662,000	(57,695,000)	35,300,000	(31,402,000)
Sales of MBS
Proceeds from sale of MBS categorized as available-for-sale		19,800,000	16,300,000	51,800,000	27,200,000
Gains on sales of MBS		9,200,000	7,600,000	19,000,000	14,100,000
Non-Agency MBS | Discount Designated as Credit Reserve and OTTI
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period		(757,564,000)	(873,533,000)	(787,541,000)	(900,557,000)
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing				0	(15,543,000)
Impact of RMBS Issuer settlement		0	0	0	0
Accretion of discount		0	0	0	0
Realized credit losses		15,729,000	21,226,000	33,779,000	40,850,000
Purchases		(6,581,000)	(711,000)	(10,875,000)	(745,000)
Sales		1,863,000	848,000	13,883,000	1,897,000
Net impairment losses recognized in earnings		0	(298,000)	0	(705,000)
Transfers/release of credit reserve		22,355,000	5,451,000	26,556,000	27,786,000
Balance at end of period	(900,557,000)	(724,198,000)	(847,017,000)	(724,198,000)	(847,017,000)
Non-Agency MBS | Accretable Discount
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period		(281,331,000)	(388,708,000)	(312,182,000)	(399,564,000)
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing				0	1,832,000
Impact of RMBS Issuer settlement		(52,881,000)	0	(52,881,000)	0
Accretion of discount		19,511,000	24,095,000	40,917,000	48,895,000
Realized credit losses		0	0	0	0
Purchases		1,774,000	(715,000)	3,380,000	(4,125,000)
Sales		9,734,000	7,833,000	21,774,000	17,802,000
Net impairment losses recognized in earnings		0	0	0	0
Transfers/release of credit reserve		(22,355,000)	(5,451,000)	(26,556,000)	(27,786,000)
Balance at end of period	$ (399,564,000)	(325,548,000)	(362,946,000)	(325,548,000)	(362,946,000)
Non-Agency MBS | Expected to Recover Par
Fair Value
Less than 12 Months		665,270,000		665,270,000
12 Months or more		724,895,000		724,895,000
Total		1,390,165,000		1,390,165,000
Unrealized Losses
Less than 12 Months		2,529,000		2,529,000
12 Months or more		5,010,000		5,010,000
Total		$ 7,539,000		$ 7,539,000
Number of Securities
Less than 12 months (in security) | Security		23		23
12 months or more (in security) | Security		23		23
Gross Unrealized Losses		$ 7,539,000		$ 7,539,000		20,143,000
Non-Agency MBS | Expected to Recover Less Than Par
Fair Value
Less than 12 Months		66,546,000		66,546,000
12 Months or more		116,340,000		116,340,000
Total		182,886,000		182,886,000
Unrealized Losses
Less than 12 Months		941,000		941,000
12 Months or more		5,121,000		5,121,000
Total		$ 6,062,000		$ 6,062,000
Number of Securities
Less than 12 months (in security) | Security		16		16
12 months or more (in security) | Security		20		20
Gross Unrealized Losses		$ 6,062,000		$ 6,062,000		8,214,000
Legacy Non-Agency MBS
Number of Securities
Gross Unrealized Losses		6,800,000		6,800,000
MBS and CRT Securities Interest Income
Coupon interest		39,549,000	46,974,000	79,858,000	96,330,000
Effective yield adjustment		19,302,000	23,454,000	39,216,000	47,860,000
Interest income		58,851,000	70,428,000	119,074,000	144,190,000
RPL/NPL MBS
Number of Securities
Gross Unrealized Losses		6,800,000		6,800,000
MBS and CRT Securities Interest Income
Coupon interest		24,773,000	21,518,000	49,142,000	40,703,000
Effective yield adjustment		141,000	565,000	1,701,000	909,000
Interest income		24,914,000	22,083,000	50,843,000	41,612,000
CRT securities
Fair Value
Less than 12 Months		51,558,000		51,558,000
12 Months or more		4,708,000		4,708,000
Total		56,266,000		56,266,000
Unrealized Losses
Less than 12 Months		296,000		296,000
12 Months or more		293,000		293,000
Total		$ 589,000		$ 589,000
Number of Securities
Less than 12 months (in security) | Security		14		14
12 months or more (in security) | Security		1		1
Gross Unrealized Losses		$ 589,000		$ 589,000		3,147,000
MBS and CRT Securities Interest Income
Coupon interest		2,807,000	1,366,000	5,163,000	2,568,000
Effective yield adjustment		415,000	158,000	751,000	316,000
Interest income		3,222,000	$ 1,524,000	5,914,000	$ 2,884,000
CRT securities | Fair value option elected, debt securities
Fair Value
Total		36,500,000		36,500,000
Number of Securities
Gross Unrealized Losses		$ 273,000		$ 273,000		$ 555,000