Schedule IV - Mortgage Loans on Real Estate (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2019 USD ($) loan
SEC Schedule, 12-29, Real Estate Companies, Investment in Movement in Mortgage Loans on Real Estate [Roll Forward]
Residential whole loans, at carrying value, beginning balance	$ 797,987
Residential whole loans, at fair value, beginning balance	1,665,978	[1]
Residential whole loans, at carrying value, ending balance	698,474
Residential whole loans, at fair value, ending balance	$ 1,381,583	[1]
Residential whole loans, at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number (loans) | loan	23,536
Balance Sheet Reported Amount	$ 7,450,709
Principal Amount of Loans Subject to Delinquent Principal or Interest	967,640
Tax basis of loans	$ 7,300,000
Residential whole loans, at fair value | Carrying value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number (loans) | loan	17,081
Balance Sheet Reported Amount	$ 6,069,126
Principal Amount of Loans Subject to Delinquent Principal or Interest	200,319
SEC Schedule, 12-29, Real Estate Companies, Investment in Movement in Mortgage Loans on Real Estate [Roll Forward]
Residential whole loans, at carrying value, beginning balance	3,016,715
Purchases and capitalized advances	4,208,603
Premium amortization/discount accretion, net	29,204
Cash collections for principal and liquidations	(1,161,375)
Provision for loan loss	(2,057)
Repurchases	(5,447)
Transfer to REO	(19,298)
Residential whole loans, at carrying value, ending balance	$ 6,066,345
Residential whole loans, at fair value | Fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number (loans) | loan	6,455
Balance Sheet Reported Amount	$ 1,381,583
Principal Amount of Loans Subject to Delinquent Principal or Interest	767,321
SEC Schedule, 12-29, Real Estate Companies, Investment in Movement in Mortgage Loans on Real Estate [Roll Forward]
Residential whole loans, at fair value, beginning balance	1,665,978
Purchases and capitalized advances	40,264
Cash collections for principal and liquidations	(152,012)
Changes in unrealized gain/(losses)	47,849
Repurchases	(1,337)
Transfer to REO	(219,159)
Residential whole loans, at fair value, ending balance	$ 1,381,583
Original loan balance $0 - $149,999 | Residential whole loans, at fair value | Carrying value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number (loans) | loan	4,783
Balance Sheet Reported Amount	$ 426,999
Principal Amount of Loans Subject to Delinquent Principal or Interest	$ 19,807
Original loan balance $0 - $149,999 | Residential whole loans, at fair value | Carrying value | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.00%
Original loan balance $0 - $149,999 | Residential whole loans, at fair value | Carrying value | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	13.08%
Original loan balance $0 - $149,999 | Residential whole loans, at fair value | Fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number (loans) | loan	2,329
Balance Sheet Reported Amount	$ 188,123
Principal Amount of Loans Subject to Delinquent Principal or Interest	$ 94,392
Original loan balance $0 - $149,999 | Residential whole loans, at fair value | Fair value | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.00%
Original loan balance $0 - $149,999 | Residential whole loans, at fair value | Fair value | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	14.99%
Original loan balance $150,000 - $299,999 | Residential whole loans, at fair value | Carrying value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number (loans) | loan	5,472
Balance Sheet Reported Amount	$ 1,059,013
Principal Amount of Loans Subject to Delinquent Principal or Interest	$ 49,221
Original loan balance $150,000 - $299,999 | Residential whole loans, at fair value | Carrying value | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.00%
Original loan balance $150,000 - $299,999 | Residential whole loans, at fair value | Carrying value | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	13.49%
Original loan balance $150,000 - $299,999 | Residential whole loans, at fair value | Fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number (loans) | loan	2,170
Balance Sheet Reported Amount	$ 393,282
Principal Amount of Loans Subject to Delinquent Principal or Interest	$ 210,176
Original loan balance $150,000 - $299,999 | Residential whole loans, at fair value | Fair value | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.92%
Original loan balance $150,000 - $299,999 | Residential whole loans, at fair value | Fair value | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	11.53%
Original loan balance $300,000 - $449,999 | Residential whole loans, at fair value | Carrying value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number (loans) | loan	3,380
Balance Sheet Reported Amount	$ 1,168,588
Principal Amount of Loans Subject to Delinquent Principal or Interest	$ 49,674
Original loan balance $300,000 - $449,999 | Residential whole loans, at fair value | Carrying value | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.90%
Original loan balance $300,000 - $449,999 | Residential whole loans, at fair value | Carrying value | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	10.50%
Original loan balance $300,000 - $449,999 | Residential whole loans, at fair value | Fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number (loans) | loan	1,364
Balance Sheet Reported Amount	$ 434,319
Principal Amount of Loans Subject to Delinquent Principal or Interest	$ 241,382
Original loan balance $300,000 - $449,999 | Residential whole loans, at fair value | Fair value | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.00%
Original loan balance $300,000 - $449,999 | Residential whole loans, at fair value | Fair value | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	10.75%
Original loan balance greater than $449,999 | Residential whole loans, at fair value | Carrying value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number (loans) | loan	3,446
Balance Sheet Reported Amount	$ 3,414,526
Principal Amount of Loans Subject to Delinquent Principal or Interest	$ 81,617
Original loan balance greater than $449,999 | Residential whole loans, at fair value | Carrying value | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.90%
Original loan balance greater than $449,999 | Residential whole loans, at fair value | Carrying value | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	11.25%
Original loan balance greater than $449,999 | Residential whole loans, at fair value | Fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number (loans) | loan	592
Balance Sheet Reported Amount	$ 365,859
Principal Amount of Loans Subject to Delinquent Principal or Interest	$ 221,371
Original loan balance greater than $449,999 | Residential whole loans, at fair value | Fair value | Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.00%
Original loan balance greater than $449,999 | Residential whole loans, at fair value | Fair value | Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	10.20%
Non-QM loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Allowance for losses	$ 388
Certain rehabilitation loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Allowance for losses	2,300
Single-family rental loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Allowance for losses	$ 62

[1] Includes approximately $186.4 million and $209.4 million of Residential whole loans, at carrying value and $567.4 million and $694.7 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at December 31, 2019 and 2018, respectively. Such assets can be used only to settle the obligations of each respective VIE.