Residential Whole Loans - Interest Income Components (Details) - Residential whole loans, total or weighted average - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2022	Jun. 30, 2021	Jun. 30, 2022	Jun. 30, 2021
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 102,354	$ 69,016	$ 201,820	$ 133,554
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	8,672	11,303	17,681	19,593
Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	74,872	38,005	144,603	75,932
Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	34,512	21,968	67,463	44,156
Performing | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	15,188	7,329	30,047	13,996
Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	16,413	6,906	29,739	13,987
Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,155	1,540	2,165	3,531
Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	7,604	262	15,189	262
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	18,810	19,708	39,536	38,029
Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	27,136	46,400	58,554	92,617
Carrying Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	8,672	11,303	17,681	19,593
Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	18,464	35,097	40,873	73,024
Carrying Value | Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	12,255	20,164	25,395	42,352
Carrying Value | Performing | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,271	6,954	4,837	13,621
Carrying Value | Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	3,783	6,439	8,476	13,520
Carrying Value | Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,155	1,540	2,165	3,531
Carrying Value | Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Carrying Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	75,218	22,616	143,266	40,937
Fair Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	56,408	2,908	103,730	2,908
Fair Value | Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	22,257	1,804	42,068	1,804
Fair Value | Performing | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	13,917	375	25,210	375
Fair Value | Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	12,630	467	21,263	467
Fair Value | Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Fair Value | Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	7,604	262	15,189	262
Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 18,810	$ 19,708	$ 39,536	$ 38,029