Residential Whole Loans - Residential Whole Loans (Details) $ in Thousands	Dec. 31, 2022 USD ($) loan	Dec. 31, 2021 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 1,826,529
Allowance for Credit Losses	(35,314)	$ (39,447)
Total Residential Whole Loans	$ 7,518,739	$ 7,913,000
Number of loans | loan	23,843	24,095
Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 6,287,650	$ 6,332,405
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	796,109	1,072,270
Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses	(35,314)	(39,447)
Total Residential Whole Loans	$ 1,791,215	$ 2,607,651
Number of loans | loan	7,126	9,361
Held at Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 1,356,235	$ 2,099,326
Held at Carrying Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses	0	0
Total Residential Whole Loans	$ 5,727,524	$ 5,305,349
Number of loans | loan	16,717	14,734
Held at Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 4,931,415	$ 4,233,079
Held at Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	796,109	1,072,270
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	987,282
Non-QM loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	3,359,830	3,461,531
Non-QM loans | Held at Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	987,282	1,448,162
Non-QM loans | Held at Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	2,372,548	2,013,369
Transitional loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	75,188
Transitional loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,417,220	734,845
Transitional loans | Held at Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	75,188	217,315
Transitional loans | Held at Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,342,032	517,530
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	210,833
Single-family rental loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,376,574	951,223
Single-family rental loans | Held at Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	210,833	331,808
Single-family rental loans | Held at Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,165,741	619,415
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	82,932
Seasoned performing loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	82,932	102,041
Seasoned performing loans | Held at Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	82,932	102,041
Seasoned performing loans | Held at Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Agency eligible investor loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	51,094	1,082,765
Agency eligible investor loans | Held at Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Agency eligible investor loans | Held at Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	51,094	1,082,765
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	470,294	547,772
Purchased Credit Deteriorated Loans | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	470,294	547,772
Purchased Credit Deteriorated Loans | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
One-to-four-family residential property loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	784,900	521,000
Multi-Family Transitional Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 632,300	$ 213,900