Use of Special Purpose Entities and Variable Interest Entities (Details) - USD ($)	3 Months Ended								9 Months Ended	12 Months Ended
	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Sep. 30, 2015	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Special purpose entities and variable interest entities
MBS and CRT securities	$ 9,969,163,000				$ 11,356,643,000					$ 9,969,163,000	$ 11,356,643,000
Residential Whole Loans	1,400,000,000				895,100,000					1,400,000,000	895,100,000
Residential whole loans, at carrying value	590,540,000				271,845,000					590,540,000	271,845,000
Residential whole loans, at fair value	$ 814,682,000				623,300,000					$ 814,682,000	623,300,000
Equity interest in certificates	100.00%									100.00%
Interest income residential whole loans									$ 16,000,000	$ 23,916,000	16,036,000	$ 4,083,000
Net gain on residential whole loans held at fair value	$ 14,632,000	$ 18,701,000	$ 14,470,000	$ 11,881,000	6,899,000	$ 5,565,000	$ 3,224,000	$ 2,034,000		59,684,000	17,722,000	$ 116,000
Non-Agency MBS
Special purpose entities and variable interest entities
MBS and CRT securities	5,825,816,000				6,420,817,000					5,825,816,000	6,420,817,000
Non-Agency MBS Transfered to Consolidated VIEs
Special purpose entities and variable interest entities
Taxable income currently estimated										$ 70,900,000
Taxable income currently estimated per share (in USD per share)										$ 0.19
Non-Agency MBS Transfered to Consolidated VIEs | Non-Agency MBS
Special purpose entities and variable interest entities
MBS and CRT securities	174,400,000				598,300,000					$ 174,400,000	598,300,000
WFMLT Series 2012-RR1
Special purpose entities and variable interest entities
Securitized debt	$ 0				$ 22,100,000					$ 0	$ 22,100,000
Trust, Ownership in Residential Whole Loans
Special purpose entities and variable interest entities
Equity interest in certificates	100.00%									100.00%