MBS and CRT Securities (Details 2)	3 Months Ended		9 Months Ended
	Sep. 30, 2015 USD ($) Security	Sep. 30, 2014 USD ($)	Sep. 30, 2015 USD ($) Security	Sep. 30, 2014 USD ($)	Dec. 31, 2014 USD ($)
Fair Value
Less than 12 Months	$ 2,388,922,000		$ 2,388,922,000
12 Months or more	1,270,196,000		1,270,196,000
Total	3,659,118,000		3,659,118,000
Unrealized Losses
Less than 12 Months	12,674,000		12,674,000
12 Months or more	21,300,000		21,300,000
Total	$ 33,974,000		$ 33,974,000
Number of Securities
Less than 12 months (in security) | Security	174		174
12 months or more (in security) | Security	193		193
Unrealized loss on investment securities	$ (33,974,000)		$ (33,974,000)		$ (39,448,000)
Total OTTI losses	0	$ 0	(525,000)	$ 0
OTTI reclassified from OCI	0	0	(180,000)	0
Net Impairment Losses Recognized in Earnings	0	0	(705,000)	0
Roll-forward of the credit loss component of OTTI
Credit loss component of OTTI at beginning of period	36,820,000		36,115,000
Additions for credit related OTTI not previously recognized	0		461,000
Subsequent additional credit related OTTI recorded	0		244,000
Credit loss component of OTTI at end of period	36,820,000		36,820,000
Changes in the components of the purchase discount on Non-Agency MBS
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing	0	0	4,537,000	0
AOCI from AFS Securities:
Unrealized gain on AFS securities at beginning of period	759,151,000	905,704,000	813,515,000	752,912,000
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing	0	0	4,537,000	0
Reclassification adjustment for MBS sales included in net income	(11,363,000)	(13,589,000)	(26,414,000)	(21,180,000)
Reclassification adjustment for OTTI included in net income	0	0	(705,000)	0
Change in AOCI from AFS securities	(55,402,000)	(56,999,000)	(109,766,000)	95,793,000
Balance at end of period	703,749,000	848,705,000	703,749,000	848,705,000
Total MBS
Fair Value
Less than 12 Months	2,299,669,000		2,299,669,000
12 Months or more	1,265,658,000		1,265,658,000
Total	3,565,327,000		3,565,327,000
Unrealized Losses
Less than 12 Months	11,389,000		11,389,000
12 Months or more	20,837,000		20,837,000
Total	$ 32,226,000		$ 32,226,000
Number of Securities
Less than 12 months (in security) | Security	155		155
12 months or more (in security) | Security	192		192
Unrealized loss on investment securities	$ (32,226,000)		$ (32,226,000)		(37,334,000)
Agency Mortgage Backed Securities
Fair Value
Less than 12 Months	768,527,000		768,527,000
12 Months or more	1,184,501,000		1,184,501,000
Total	1,953,028,000		1,953,028,000
Unrealized Losses
Less than 12 Months	5,290,000		5,290,000
12 Months or more	17,530,000		17,530,000
Total	$ 22,820,000		$ 22,820,000
Number of Securities
Less than 12 months (in security) | Security	98		98
12 months or more (in security) | Security	173		173
Unrealized loss on investment securities	$ (22,820,000)		$ (22,820,000)		(33,622,000)
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	(2,028,000)	(14,937,000)	(14,393,000)	46,000,000
MBS and CRT Securities Interest Income
Coupon interest	35,154,000	46,529,000	113,543,000	145,864,000
Effective yield adjustment	(11,536,000)	(13,463,000)	(32,513,000)	(35,860,000)
Interest income	23,618,000	33,066,000	81,030,000	110,004,000
Agency Mortgage Backed Securities | Fannie Mae
Fair Value
Less than 12 Months	508,889,000		508,889,000
12 Months or more	852,198,000		852,198,000
Total	1,361,087,000		1,361,087,000
Unrealized Losses
Less than 12 Months	2,921,000		2,921,000
12 Months or more	11,841,000		11,841,000
Total	$ 14,762,000		$ 14,762,000
Number of Securities
Less than 12 months (in security) | Security	63		63
12 months or more (in security) | Security	109		109
Unrealized loss on investment securities	$ (14,762,000)		$ (14,762,000)		(21,100,000)
Agency Mortgage Backed Securities | Freddie Mac
Fair Value
Less than 12 Months	259,638,000		259,638,000
12 Months or more	332,303,000		332,303,000
Total	591,941,000		591,941,000
Unrealized Losses
Less than 12 Months	2,369,000		2,369,000
12 Months or more	5,689,000		5,689,000
Total	$ 8,058,000		$ 8,058,000
Number of Securities
Less than 12 months (in security) | Security	35		35
12 months or more (in security) | Security	64		64
Unrealized loss on investment securities	$ (8,058,000)		$ (8,058,000)		(12,522,000)
Non-Agency MBS
Fair Value
Less than 12 Months	1,531,142,000		1,531,142,000
12 Months or more	81,157,000		81,157,000
Total	1,612,299,000		1,612,299,000
Unrealized Losses
Less than 12 Months	6,099,000		6,099,000
12 Months or more	3,307,000		3,307,000
Total	$ 9,406,000		$ 9,406,000
Number of Securities
Less than 12 months (in security) | Security	57		57
12 months or more (in security) | Security	19		19
Unrealized loss on investment securities	$ (9,406,000)		$ (9,406,000)		(3,712,000)
Credit-related OTTI losses	0		(705,000)
Changes in the components of the purchase discount on Non-Agency MBS
MBS linked transactions purchase discount designated as credit reserve reallocated to accretable purchase discount		333,000		218,000
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	(42,011,000)	(28,473,000)	(72,791,000)	70,973,000
Sales of MBS
Proceeds from sale of MBS categorized as available-for-sale	23,500,000	61,600,000	50,700,000	103,600,000
Gains on sales of MBS	11,200,000	13,900,000	25,200,000	25,300,000
Non-Agency MBS | Discount Designated as Credit Reserve and OTTI
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period	(847,017,000)	(986,842,000)	(900,557,000)	(1,043,037,000)
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing			(15,543,000)	0
Accretion of discount	0	0	0	0
Realized credit losses	21,527,000	20,733,000	62,377,000	69,129,000
Purchases	(455,000)	(4,200,000)	(1,200,000)	(70,535,000)
Sales	3,676,000	21,024,000	5,573,000	34,780,000
Net impairment losses recognized in earnings			(705,000)	0
Transfers/release of credit reserve	6,872,000	20,185,000	34,658,000	80,563,000
Balance at end of period	(815,397,000)	(929,100,000)	(815,397,000)	(929,100,000)
AOCI from AFS Securities:
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing			(15,543,000)	0
Non-Agency MBS | Accretable Discount
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period	(362,946,000)	(436,111,000)	(399,564,000)	(460,039,000)
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing			1,832,000	0
Accretion of discount	22,805,000	25,504,000	71,700,000	78,701,000
Realized credit losses	0	0	0	0
Purchases	113,000	272,000	(4,012,000)	25,314,000
Sales	11,193,000	4,169,000	28,995,000	10,236,000
Net impairment losses recognized in earnings			0	0
Transfers/release of credit reserve	(6,872,000)	(20,185,000)	(34,658,000)	(80,563,000)
Balance at end of period	(335,707,000)	(426,351,000)	(335,707,000)	(426,351,000)
AOCI from AFS Securities:
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing			1,832,000	0
Non-Agency MBS | Expected to Recover Par
Fair Value
Less than 12 Months	1,411,102,000		1,411,102,000
12 Months or more	13,249,000		13,249,000
Total	1,424,351,000		1,424,351,000
Unrealized Losses
Less than 12 Months	4,117,000		4,117,000
12 Months or more	736,000		736,000
Total	$ 4,853,000		$ 4,853,000
Number of Securities
Less than 12 months (in security) | Security	33		33
12 months or more (in security) | Security	8		8
Unrealized loss on investment securities	$ (4,853,000)		$ (4,853,000)		(1,052,000)
Non-Agency MBS | Expected to Recover Less Than Par
Fair Value
Less than 12 Months	120,040,000		120,040,000
12 Months or more	67,908,000		67,908,000
Total	187,948,000		187,948,000
Unrealized Losses
Less than 12 Months	1,982,000		1,982,000
12 Months or more	2,571,000		2,571,000
Total	$ 4,553,000		$ 4,553,000
Number of Securities
Less than 12 months (in security) | Security	24		24
12 months or more (in security) | Security	11		11
Unrealized loss on investment securities	$ (4,553,000)		$ (4,553,000)		(2,660,000)
Legacy Non-Agency MBS
MBS and CRT Securities Interest Income
Coupon interest	44,824,000	52,270,000	141,152,000	161,318,000
Effective yield adjustment	21,955,000	25,477,000	69,816,000	78,561,000
Interest income	66,779,000	77,747,000	210,968,000	239,879,000
RPL/NPL MBS
MBS and CRT Securities Interest Income
Coupon interest	22,898,000	97,000	63,602,000	800,000
Effective yield adjustment	753,000	0	1,662,000	0
Interest income	23,651,000	97,000	65,264,000	800,000
CRT securities
Fair Value
Less than 12 Months	89,253,000		89,253,000
12 Months or more	4,538,000		4,538,000
Total	93,791,000		93,791,000
Unrealized Losses
Less than 12 Months	1,285,000		1,285,000
12 Months or more	463,000		463,000
Total	$ 1,748,000		$ 1,748,000
Number of Securities
Less than 12 months (in security) | Security	19		19
12 months or more (in security) | Security	1		1
Unrealized loss on investment securities	$ (1,748,000)		$ (1,748,000)		$ (2,114,000)
MBS and CRT Securities Interest Income
Coupon interest	1,432,000	30,000	4,000,000	30,000
Effective yield adjustment	161,000	0	477,000	0
Interest income	1,593,000	$ 30,000	4,477,000	$ 30,000
CRT securities | Fair value option elected, debt securities
Fair Value
Total	9,600,000		9,600,000
Number of Securities
Unrealized loss on investment securities	$ (19,000)		$ (19,000)