Residential Whole Loans - Residential Whole Loans (Details) $ in Thousands	Dec. 31, 2021 USD ($) loan	Dec. 31, 2020 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 2,647,099
Allowance for Credit Losses	(39,447)	$ (86,833)
Total Residential Whole Loans	$ 7,913,000	$ 5,325,401
Number of loans | loan	24,095	18,734
Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 6,332,405	$ 3,521,624
Nonperforming Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,072,270	1,216,902
Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses	(39,447)	(86,833)
Total Residential Whole Loans	$ 2,607,651	$ 4,108,499
Number of loans | loan	9,361	13,112
Held at Carrying Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 2,099,326	$ 3,521,624
Held at Carrying Value | Nonperforming Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses	0	0
Total Residential Whole Loans	$ 5,305,349	$ 1,216,902
Number of loans | loan	14,734	5,622
Held at Fair Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 4,233,079	$ 0
Held at Fair Value | Nonperforming Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,072,270	1,216,902
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,448,162
Non-QM loans | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	3,461,531	2,357,185
Non-QM loans | Held at Carrying Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,448,162	2,357,185
Non-QM loans | Held at Fair Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	2,013,369	0
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	217,315
Rehabilitation loans | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	734,845	581,801
Rehabilitation loans | Held at Carrying Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	217,315	581,801
Rehabilitation loans | Held at Fair Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	517,530	0
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	331,808
Single-family rental loans | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	951,223	446,374
Single-family rental loans | Held at Carrying Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	331,808	446,374
Single-family rental loans | Held at Fair Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	619,415	0
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	102,042
Seasoned performing loans | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	102,041	136,264
Seasoned performing loans | Held at Carrying Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	102,041	136,264
Seasoned performing loans | Held at Fair Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Agency eligible investor loans | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,082,765	0
Agency eligible investor loans | Held at Carrying Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Agency eligible investor loans | Held at Fair Value | Performing Financial Instruments
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,082,765	0
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	547,772	673,708
Purchased Credit Deteriorated Loans | Held at Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	547,772	673,708
Purchased Credit Deteriorated Loans | Held at Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 0	$ 0