Securitized Debt - Narrative (Details) - USD ($)	3 Months Ended								12 Months Ended
	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Debt Instrument [Line Items]
Gains (losses) on extinguishment of debt	$ 919,000	$ (55,794,000)	$ 459,000	$ 0	$ 9,926,000	$ 0	$ (608,000)	$ 0	$ (54,418,000)	$ 9,318,000	$ 26,376,000
Non-Agency RMBS
Debt Instrument [Line Items]
Gains (losses) on extinguishment of debt		$ (55,000,000)
Securitized Loans | Non-Agency RMBS
Debt Instrument [Line Items]
Principal balance					2,900,000					2,900,000
Acquired securitized debt collateral outstanding principal balance									0	0
Repurchase of non retained secured debt										3,500,000
Gains (losses) on extinguishment of debt										608,000
Securitized Loans | Non-Agency RMBS | Variable Interest Entities, Primary Beneficiary
Debt Instrument [Line Items]
Principal balance	$ 135,000,000				152,000,000				$ 135,000,000	152,000,000
Weighted average cost of financing	6.50%								6.50%
Acquired securitized debt collateral outstanding principal balance									$ 785,000,000
Repurchase of non retained secured debt									784,000,000
Gains (losses) on extinguishment of debt									1,000,000
Securitized Loans | Securitized debt | Variable Interest Entities, Primary Beneficiary
Debt Instrument [Line Items]
Principal balance	$ 8,700,000,000				$ 8,200,000,000				$ 8,700,000,000	$ 8,200,000,000
Weighted average cost of financing	3.40%				4.20%				3.40%	4.20%