Use of Special Purpose Entities and Variable Interest Entities (Details) - USD ($)		3 Months Ended		6 Months Ended
		Jun. 30, 2016	Jun. 30, 2015	Jun. 30, 2016	Jun. 30, 2015	Dec. 31, 2015
Variable Interest Entity [Line Items]
Available-for-sale securities		$ 10,685,331,000		$ 10,685,331,000		$ 11,356,643,000
Residential whole loans		1,100,000,000		1,100,000,000		895,100,000
Residential whole loans at carrying value		392,172,000		392,172,000		271,845,000
Residential whole loans, at fair value		$ 684,582,000		$ 684,582,000		623,276,000
Ownership percentage by parent		100.00%		100.00%
Interest income residential whole loans held at carrying value		$ 5,758,000	$ 4,193,000	$ 10,195,000	$ 7,784,000
Net gain on residential whole loans held at fair value		14,470,000	$ 3,224,000	26,351,000	$ 5,258,000
Non-Agency MBS
Variable Interest Entity [Line Items]
Available-for-sale securities		6,104,880,000		6,104,880,000		6,420,817,000
Non-Agency MBS | Recurring basis
Variable Interest Entity [Line Items]
Available-for-sale securities		6,104,880,000		6,104,880,000		6,420,817,000
Non-Agency MBS | Recurring basis | Level 3
Variable Interest Entity [Line Items]
Available-for-sale securities		0		0		0
Residential Whole Loans | Recurring basis
Variable Interest Entity [Line Items]
Residential whole loans, at fair value		684,582,000		684,582,000		623,276,000
Residential Whole Loans | Recurring basis | Level 3
Variable Interest Entity [Line Items]
Residential whole loans, at fair value		684,582,000		684,582,000		623,276,000
WFMLT Series 2012-RR1
Variable Interest Entity [Line Items]
Outstanding amount of Senior Bonds at June 30, 2016		0		0		22,100,000
Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Taxable income from unwind of resecuritization transaction				$ 70,900,000
Taxable income from unwind of resecuritization transaction (in dollars per share)				$ 0.19
Variable Interest Entity, Primary Beneficiary | Non-Agency MBS
Variable Interest Entity [Line Items]
Available-for-sale securities	[1]	$ 193,700,000		$ 193,700,000		$ 598,300,000
Trust, ownership in residential whole loans
Variable Interest Entity [Line Items]
Ownership percentage by parent		100.00%		100.00%

[1] Non-Agency MBS transferred to consolidated VIEs represent assets of the consolidated VIEs that can be used only to settle the obligations of each respective VIE.