Residential Whole Loans - Residential Whole Loans, at Carrying and Fair Value (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2024 USD ($) loan	Dec. 31, 2023 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 1,484,370
Allowance for Credit Losses	(19,612)	$ (20,451)
Total Residential Whole Loans	$ 9,118,665	$ 9,041,292
Number of loans | loan	25,709	25,401
Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses	$ (19,612)	$ (20,451)
Total Residential Whole Loans	$ 1,464,758	$ 1,529,784
Number of loans | loan	6,148	6,326
Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses	$ 0	$ 0
Total Residential Whole Loans	$ 7,653,907	$ 7,511,508
Number of loans | loan	19,561	19,075
Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 8,032,841	$ 7,926,593
Performing | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,060,723	1,120,509
Performing | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	6,972,118	6,806,084
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	681,789	705,424
Nonperforming | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Nonperforming | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	681,789	705,424
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	816,617
Non-QM loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	3,838,386	3,805,577
Non-QM loans | Performing | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	816,617	843,884
Non-QM loans | Performing | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	3,021,769	2,961,693
Transitional loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	2,494,772	2,361,496
Transitional loans | Performing | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	29,098	35,467
Transitional loans | Performing | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	2,465,674	2,326,029
Single-family Rental Loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,578,964	1,634,796
Loans held for sale	0	13,600
Gain from sale of loans	500
Single-family Rental Loans | Performing | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	148,943	172,213
Single-family Rental Loans | Performing | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,430,021	1,462,583
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	66,065
Seasoned performing loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	66,065	68,945
Seasoned performing loans | Performing | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	66,065	68,945
Seasoned performing loans | Performing | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Agency eligible investor loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	54,654	55,779
Agency eligible investor loans | Performing | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Agency eligible investor loans | Performing | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	54,654	55,779
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	423,647	429,726
Purchased Credit Deteriorated Loans | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	423,647	429,726
Purchased Credit Deteriorated Loans | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Single Family Residential Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,300,000	1,200,000
Multi-Family Transitional Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,200,000	1,200,000
New Construction Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 506,500	$ 471,100