Residential Whole Loans (Details) $ in Thousands	3 Months Ended
	Dec. 31, 2022 USD ($) loan	Mar. 31, 2023 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		$ 1,753,080
Allowance for Credit Losses	$ (35,314)	(33,061)
Total Residential Whole Loans	$ 7,518,739	$ 7,794,250
Number of loans | loan	23,843	24,052
Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale	$ 337,800
Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses	(35,314)	$ (33,061)
Total Residential Whole Loans	$ 1,791,215	$ 1,720,018
Number of loans | loan	7,126	6,930
Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses	$ 0	$ 0
Total Residential Whole Loans	$ 5,727,524	$ 6,074,232
Number of loans | loan	16,717	17,122
Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 6,287,650	$ 6,591,264
Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,356,235	1,292,399
Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	4,931,415	5,298,865
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	796,109	775,367
Nonperforming | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, sale	598,000
Nonperforming | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Nonperforming | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	796,109	775,367
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		958,099
Non-QM loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	3,359,830	3,459,231
Non-QM loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	987,282	958,099
Non-QM loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	2,372,548	2,501,132
Transitional loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,417,220	1,524,905
Transitional loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	75,188	53,272
Transitional loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,342,032	1,471,633
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		201,563
Single-family rental loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,376,574	1,466,809
Single-family rental loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	210,833	201,563
Single-family rental loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	1,165,741	1,265,246
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		79,465
Seasoned performing loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	82,932	79,465
Seasoned performing loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	82,932	79,465
Seasoned performing loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Agency eligible investor loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	51,094	60,854
Agency eligible investor loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Agency eligible investor loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	51,094	60,854
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	470,294	460,680
Purchased Credit Deteriorated Loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	470,294	460,680
Purchased Credit Deteriorated Loans | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	0	0
Single Family Residential Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	784,900	825,900
Multi-Family Transitional Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans	$ 632,300	$ 699,000