Fair Value of Financial Instruments - Level 3 Assets (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2021	Dec. 31, 2020
Level 2
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Transfer to Level 2	$ 654,700
Recurring basis | Level 3 | Residential whole loans, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	1,216,902	$ 1,381,583
Purchases and originations	4,367,423	0
Draws	53,599	0
Changes in fair value recorded in Net gain on residential whole loans measured at fair value through earnings	16,243	17,204
Repayments	(295,790)	(92,733)
Loan sales and repurchases	(2,023)	(18,530)
Transfer to REO	(51,005)	(70,622)
Transfer to Level 2	(1,082,765)	0
Balance at end of period	$ 4,222,584	$ 1,216,902