Residential Whole Loans - Residential Whole Loans (Details) $ in Thousands	Mar. 31, 2022 USD ($) loan	Dec. 31, 2021 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 2,320,048
Allowance for Credit Losses	(35,457)	$ (39,447)
Total Residential Whole Loans	$ 8,261,906	$ 7,913,000
Number of loans | loan	25,212	24,095
Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses	$ (35,457)	$ (39,447)
Total Residential Whole Loans	$ 2,284,591	$ 2,607,651
Number of loans | loan	8,506	9,361
Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses	$ 0	$ 0
Total Residential Whole Loans	$ 5,977,315	$ 5,305,349
Number of loans | loan	16,706	14,734
Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 6,791,119	$ 6,332,405
Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,801,598	2,099,326
Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	4,989,521	4,233,079
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	987,794	1,072,270
Nonperforming | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Nonperforming | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	987,794	1,072,270
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,265,731
Non-QM loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	3,657,363	3,461,531
Non-QM loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,265,731	1,448,162
Non-QM loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	2,391,632	2,013,369
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	154,508
Rehabilitation loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	890,357	734,845
Rehabilitation loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	154,508	217,315
Rehabilitation loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	735,849	517,530
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	283,090
Single-family rental loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,153,497	951,223
Single-family rental loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	283,090	331,808
Single-family rental loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	870,407	619,415
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	98,269
Seasoned performing loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	98,269	102,041
Seasoned performing loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	98,269	102,041
Seasoned performing loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Agency eligible investor loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	991,633	1,082,765
Agency eligible investor loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Agency eligible investor loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	991,633	1,082,765
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	518,450	547,772
Purchased Credit Deteriorated Loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	518,450	547,772
Purchased Credit Deteriorated Loans | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 0	$ 0