MBS and CRT Securities (Details 2)	3 Months Ended		6 Months Ended
	Jun. 30, 2015 USD ($) Security	Jun. 30, 2014 USD ($)	Jun. 30, 2015 USD ($) Security	Jun. 30, 2014 USD ($)	Dec. 31, 2014 USD ($)
Fair Value
Less than 12 Months	$ 1,714,759,000		$ 1,714,759,000
12 Months or more	1,285,953,000		1,285,953,000
Total	3,000,712,000		3,000,712,000
Unrealized Losses
Less than 12 Months	13,090,000		13,090,000
12 Months or more	26,496,000		26,496,000
Total	$ 39,586,000		$ 39,586,000
Number of Securities
Less than 12 months (in security) | Security	139		139
12 months or more (in security) | Security	188		188
Unrealized loss on investment securities	$ (39,586,000)		$ (39,586,000)		$ (39,448,000)
Total OTTI losses	(130,000)	$ 0	(525,000)	$ 0
OTTI reclassified from OCI	(168,000)	0	(180,000)	0
OTTI recognized in earnings	(298,000)	0	(705,000)	0
Roll-forward of the credit loss component of OTTI
Credit loss component of OTTI at beginning of period	36,522,000		36,115,000
Additions for credit related OTTI not previously recognized	66,000		461,000
Subsequent additional credit related OTTI recorded	232,000		244,000
Credit loss component of OTTI at end of period	36,820,000		36,820,000
Changes in the components of the purchase discount on Non-Agency MBS
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing	0	0	4,537,000	0
AOCI from AFS Securities:
Unrealized gain on AFS securities at beginning of period	850,257,000	819,222,000	813,515,000	752,912,000
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing	0	0	4,537,000	0
Reclassification adjustment for MBS sales included in net income	(7,863,000)	(6,748,000)	(14,429,000)	(9,699,000)
Reclassification adjustment for OTTI included in net income	(298,000)	0	(705,000)	0
Change in AOCI from AFS securities	(91,106,000)	86,482,000	(54,364,000)	152,792,000
Balance at end of period	759,151,000	905,704,000	759,151,000	905,704,000
Total MBS
Fair Value
Less than 12 Months	1,684,082,000		1,684,082,000
12 Months or more	1,281,303,000		1,281,303,000
Total	2,965,385,000		2,965,385,000
Unrealized Losses
Less than 12 Months	12,945,000		12,945,000
12 Months or more	26,147,000		26,147,000
Total	$ 39,092,000		$ 39,092,000
Number of Securities
Less than 12 months (in security) | Security	134		134
12 months or more (in security) | Security	187		187
Unrealized loss on investment securities	$ (39,092,000)		$ (39,092,000)		(37,334,000)
Agency Mortgage Backed Securities
Fair Value
Less than 12 Months	793,329,000		793,329,000
12 Months or more	1,235,039,000		1,235,039,000
Total	2,028,368,000		2,028,368,000
Unrealized Losses
Less than 12 Months	8,981,000		8,981,000
12 Months or more	24,465,000		24,465,000
Total	$ 33,446,000		$ 33,446,000
Number of Securities
Less than 12 months (in security) | Security	97		97
12 months or more (in security) | Security	172		172
Unrealized loss on investment securities	$ (33,446,000)		$ (33,446,000)		(33,622,000)
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	(25,250,000)	43,094,000	(12,365,000)	60,937,000
MBS and CRT Securities Interest Income
Coupon interest	37,646,000	49,836,000	78,388,000	99,335,000
Effective yield adjustment	(11,907,000)	(12,227,000)	(20,976,000)	(22,397,000)
Interest income	25,739,000	37,609,000	57,412,000	76,938,000
Agency Mortgage Backed Securities | Fannie Mae
Fair Value
Less than 12 Months	488,886,000		488,886,000
12 Months or more	891,047,000		891,047,000
Total	1,379,933,000		1,379,933,000
Unrealized Losses
Less than 12 Months	4,232,000		4,232,000
12 Months or more	16,082,000		16,082,000
Total	$ 20,314,000		$ 20,314,000
Number of Securities
Less than 12 months (in security) | Security	57		57
12 months or more (in security) | Security	108		108
Unrealized loss on investment securities	$ (20,314,000)		$ (20,314,000)		(21,100,000)
Agency Mortgage Backed Securities | Freddie Mac
Fair Value
Less than 12 Months	304,443,000		304,443,000
12 Months or more	343,992,000		343,992,000
Total	648,435,000		648,435,000
Unrealized Losses
Less than 12 Months	4,749,000		4,749,000
12 Months or more	8,383,000		8,383,000
Total	$ 13,132,000		$ 13,132,000
Number of Securities
Less than 12 months (in security) | Security	40		40
12 months or more (in security) | Security	64		64
Unrealized loss on investment securities	$ (13,132,000)		$ (13,132,000)		(12,522,000)
Non-Agency MBS
Fair Value
Less than 12 Months	890,753,000		890,753,000
12 Months or more	46,264,000		46,264,000
Total	937,017,000		937,017,000
Unrealized Losses
Less than 12 Months	3,964,000		3,964,000
12 Months or more	1,682,000		1,682,000
Total	$ 5,646,000		$ 5,646,000
Number of Securities
Less than 12 months (in security) | Security	37		37
12 months or more (in security) | Security	15		15
Unrealized loss on investment securities	$ (5,646,000)		$ (5,646,000)		(3,712,000)
Credit-related OTTI losses	(298,000)		(705,000)
Changes in the components of the purchase discount on Non-Agency MBS
MBS linked transactions purchase discount designated as credit reserve reallocated to accretable purchase discount		0		115,000
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	(57,695,000)	50,136,000	(31,402,000)	101,554,000
Sales of MBS
Proceeds from sale of MBS categorized as available-for-sale	16,300,000	26,500,000	27,200,000	42,000,000
Gains on sales of MBS	7,600,000	7,900,000	14,100,000	11,400,000
Non-Agency MBS | Discount Designated as Credit Reserve and OTTI
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period	(873,533,000)	(1,041,933,000)	(900,557,000)	(1,043,037,000)
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing			(15,543,000)	0
Accretion of discount	0	0	0	0
Realized credit losses	21,226,000	23,359,000	40,850,000	48,396,000
Purchases	(711,000)	(3,018,000)	(745,000)	(66,335,000)
Sales	848,000	10,269,000	1,897,000	13,756,000
Net impairment losses recognized in earnings	(298,000)	0	(705,000)	0
Transfers/release of credit reserve	5,451,000	24,481,000	27,786,000	60,378,000
Balance at end of period	(847,017,000)	(986,842,000)	(847,017,000)	(986,842,000)
AOCI from AFS Securities:
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing			(15,543,000)	0
Non-Agency MBS | Accretable Discount
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period	(388,708,000)	(442,156,000)	(399,564,000)	(460,039,000)
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing			1,832,000	0
Accretion of discount	24,095,000	25,766,000	48,895,000	53,197,000
Realized credit losses	0	0	0	0
Purchases	(715,000)	1,636,000	(4,125,000)	25,042,000
Sales	7,833,000	3,124,000	17,802,000	6,067,000
Net impairment losses recognized in earnings	0	0	0	0
Transfers/release of credit reserve	(5,451,000)	(24,481,000)	(27,786,000)	(60,378,000)
Balance at end of period	(362,946,000)	(436,111,000)	(362,946,000)	(436,111,000)
AOCI from AFS Securities:
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing			1,832,000	0
Non-Agency MBS | Expected to Recover Par
Fair Value
Less than 12 Months	703,830,000		703,830,000
12 Months or more	16,734,000		16,734,000
Total	720,564,000		720,564,000
Unrealized Losses
Less than 12 Months	1,473,000		1,473,000
12 Months or more	787,000		787,000
Total	$ 2,260,000		$ 2,260,000
Number of Securities
Less than 12 months (in security) | Security	14		14
12 months or more (in security) | Security	8		8
Unrealized loss on investment securities	$ (2,260,000)		$ (2,260,000)		(1,052,000)
Non-Agency MBS | Expected to Recover Less Than Par
Fair Value
Less than 12 Months	186,923,000		186,923,000
12 Months or more	29,530,000		29,530,000
Total	216,453,000		216,453,000
Unrealized Losses
Less than 12 Months	2,491,000		2,491,000
12 Months or more	895,000		895,000
Total	$ 3,386,000		$ 3,386,000
Number of Securities
Less than 12 months (in security) | Security	23		23
12 months or more (in security) | Security	7		7
Unrealized loss on investment securities	$ (3,386,000)		$ (3,386,000)		(2,660,000)
Legacy Non-Agency MBS
MBS and CRT Securities Interest Income
Coupon interest	46,974,000	55,289,000	96,330,000	109,678,000
Effective yield adjustment	23,454,000	25,694,000	47,860,000	53,083,000
Interest income	70,428,000	80,983,000	144,190,000	162,761,000
RPL/NPL MBS
MBS and CRT Securities Interest Income
Coupon interest	21,518,000	33,000	40,703,000	74,000
Effective yield adjustment	565,000	0	909,000	0
Interest income	22,083,000	33,000	41,612,000	74,000
CRT securities
Fair Value
Less than 12 Months	30,677,000		30,677,000
12 Months or more	4,650,000		4,650,000
Total	35,327,000		35,327,000
Unrealized Losses
Less than 12 Months	145,000		145,000
12 Months or more	349,000		349,000
Total	$ 494,000		$ 494,000
Number of Securities
Less than 12 months (in security) | Security	5		5
12 months or more (in security) | Security	1		1
Unrealized loss on investment securities	$ (494,000)		$ (494,000)		$ (2,114,000)
MBS and CRT Securities Interest Income
Coupon interest	1,366,000	0	2,568,000	0
Effective yield adjustment	158,000	0	316,000	0
Interest income	$ 1,524,000	$ 0	$ 2,884,000	$ 0