Residential Whole Loans - Interest Income Components (Details) - Residential Whole Loans - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2022	Dec. 31, 2021	Dec. 31, 2020
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 441,223	$ 303,468	$ 332,212
Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	329,276	182,597	222,526
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	78,520	80,741	73,448
Non-QM loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	149,743	96,948	136,527
Transitional loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	75,524	33,129	49,484
Single-family rental loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	68,975	34,169	27,722
Seasoned performing loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	4,673	6,684	8,793
Agency eligible investor loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	30,361	11,667	0
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	33,427	40,130	36,238
Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	112,583	169,618	258,764
Held at Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	79,156	129,488	222,526
Held at Carrying Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0
Held at Carrying Value | Non-QM loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	51,359	75,517	136,527
Held at Carrying Value | Transitional loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	7,810	22,424	49,484
Held at Carrying Value | Single-family rental loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	15,314	24,863	27,722
Held at Carrying Value | Seasoned performing loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	4,673	6,684	8,793
Held at Carrying Value | Agency eligible investor loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0
Held at Carrying Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	33,427	40,130	36,238
Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	328,640	133,850	73,448
Held at Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	250,120	53,109	0
Held at Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	78,520	80,741	73,448
Held at Fair Value | Non-QM loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	98,384	21,431	0
Held at Fair Value | Transitional loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	67,714	10,705	0
Held at Fair Value | Single-family rental loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	53,661	9,306	0
Held at Fair Value | Seasoned performing loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0
Held at Fair Value | Agency eligible investor loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	30,361	11,667	0
Held at Fair Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 0	$ 0	$ 0