Use of Special Purpose Entities and Variable Interest Entities (Details) - USD ($)		3 Months Ended								12 Months Ended
		Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Variable Interest Entity [Line Items]
Unpaid principal balance residential whole loans sold		$ 1,325,115,000				$ 814,682,000				$ 1,325,115,000	$ 814,682,000
Residential whole loans, at carrying value	[1]	908,516,000				590,540,000				908,516,000	590,540,000
Residential whole loans, at fair value	[1]	1,325,115,000				814,682,000				1,325,115,000	814,682,000
Securitized debt		363,944,000				0				363,944,000	0
MBS and CRT securities		7,023,050,000				9,828,183,000				7,023,050,000	9,828,183,000
Residential whole loans		2,200,000,000				1,400,000,000				2,200,000,000	1,400,000,000
Interest income residential whole loans										36,187,000	23,916,000	$ 16,036,000
Net gain on residential whole loans held at fair value		41,359,000	$ 18,679,000	$ 16,208,000	$ 13,773,000	14,876,000	$ 19,639,000	$ 15,742,000	$ 12,348,000	90,019,000	62,605,000	$ 19,575,000
Non-Agency MBS
Variable Interest Entity [Line Items]
MBS and CRT securities		3,533,966,000				5,684,836,000				3,533,966,000	5,684,836,000
Non-Agency MBS Transfered to Consolidated VIEs
Variable Interest Entity [Line Items]
Taxable income currently estimated										$ 53,300,000
Taxable income currently estimated per share (in USD per share)										$ 0.13
Non-Agency MBS Transfered to Consolidated VIEs | Non-Agency MBS
Variable Interest Entity [Line Items]
MBS and CRT securities		0				$ 174,400,000				$ 0	$ 174,400,000
Asset-backed Securities, Securitized Loans and Receivables
Variable Interest Entity [Line Items]
Unpaid principal balance residential whole loans sold		620,900,000								620,900,000
Residential whole loans at carrying value sold		193,300,000								193,300,000
Fair value of mortgages		296,500,000								296,500,000
Residential whole loans, at carrying value		183,200,000								183,200,000
Residential whole loans, at fair value		289,300,000								289,300,000
Securitized debt		363,900,000								363,900,000
Bonds | Asset-backed Securities, Securitized Loans and Receivables | Trust, Ownership in Residential Whole Loans
Variable Interest Entity [Line Items]
Face amount		$ 382,800,000.0								$ 382,800,000.0
Weighted average interest rate		3.12%								3.12%
Rated and Non-Rated Certificates | Asset-backed Securities, Securitized Loans and Receivables | Trust, Ownership in Residential Whole Loans
Variable Interest Entity [Line Items]
Face amount		$ 127,000,000.0								$ 127,000,000.0
Proceeds from issuance of debt										382,800,000
Other Assets | Asset-backed Securities, Securitized Loans and Receivables
Variable Interest Entity [Line Items]
Residential whole loans, at fair value		$ 5,500,000								$ 5,500,000

[1] Includes approximately $183.2 million of Residential whole loans, at carrying value and $289.3 million of Residential whole loans, at fair value transferred to consolidated VIEs at December 31, 2017. Such assets can be used only to settle the obligations of each respective VIE.