Use of Special Purpose Entities and Variable Interest Entities (Details) - USD ($) $ / shares in Units, $ in Thousands		1 Months Ended	3 Months Ended
		Feb. 29, 2012	Mar. 31, 2016	Mar. 31, 2015	Dec. 31, 2015
Variable Interest Entity [Line Items]
Outstanding amount of Senior Bonds at March 31, 2016	[1]		$ 11,821		$ 21,868
Available-for-sale securities			10,861,443		11,356,643
Residential whole loans			1,024,000		895,100
Residential whole loans at carrying value			376,235		271,845
Residential whole loans, at fair value			$ 647,360		623,276
Ownership percentage by parent			100.00%
Interest income residential whole loans held at carrying value			$ 4,496	$ 3,591
Net gain on residential whole loans held at fair value			11,881	$ 2,034
Non-Agency MBS
Variable Interest Entity [Line Items]
Available-for-sale securities			6,100,757		6,420,817
Non-Agency MBS | Recurring basis
Variable Interest Entity [Line Items]
Available-for-sale securities			6,100,757		6,420,817
Non-Agency MBS | Recurring basis | Level 3
Variable Interest Entity [Line Items]
Available-for-sale securities			0		0
Residential Whole Loans | Recurring basis
Variable Interest Entity [Line Items]
Residential whole loans, at fair value			647,360		623,276
Residential Whole Loans | Recurring basis | Level 3
Variable Interest Entity [Line Items]
Residential whole loans, at fair value			647,360		623,276
WFMLT Series 2012-RR1
Variable Interest Entity [Line Items]
Principal value of Non-Agency MBS sold		$ 433,347
Face amount of Bonds issued by the VIE and purchased by 3rd party investors		$ 186,691
Outstanding amount of Senior Bonds at March 31, 2016			11,924		22,100
Pass-through rate for Senior Bonds issued, percentage		2.85%
Cash received		$ 186,691
Notional amount acquired of non-rated, interest only senior certificates		$ 186,691
Unamortized deferred costs			103
WFMLT Series 2012-RR1 | Senior-support certificates
Variable Interest Entity [Line Items]
Face amount of Senior Support Certificates received by the Company			216,603
Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Taxable income from unwind of resecuritization transaction			$ 70,900
Taxable income from unwind of resecuritization transaction (in dollars per share)			$ 0.19
Variable Interest Entity, Primary Beneficiary | Non-Agency MBS
Variable Interest Entity [Line Items]
Available-for-sale securities	[2]		$ 198,500		$ 598,300
Trust, ownership in residential whole loans
Variable Interest Entity [Line Items]
Ownership percentage by parent			100.00%

[1] Securitized debt represents third-party liabilities of consolidated VIEs and excludes liabilities of the VIEs acquired by the Company that eliminate on consolidation. The third-party beneficial interest holders in the VIEs have no recourse to the general credit of the Company. (See Notes 14 and 19 for further discussion.)

[2] Non-Agency MBS transferred to consolidated VIEs represent assets of the consolidated VIEs that can be used only to settle the obligations of each respective VIE.