Residential Whole Loans - Additional Credit Related Information (Details) - USD ($) $ in Thousands	3 Months Ended				6 Months Ended	12 Months Ended
	Jun. 30, 2023	Mar. 31, 2023	Jun. 30, 2022	Mar. 31, 2022	Jun. 30, 2023	Dec. 31, 2022
Financing Receivable, Credit Quality Indicator [Line Items]
2023	$ 0				$ 0
2022	0				0
2021	60,324				60,324
2020	215,654				215,654
2019	571,798				571,798
Prior	823,012				823,012
Residential whole loans, total or weighted average	1,670,788				1,670,788
Gross write-offs, 2023					0
Gross write-offs, 2022					0
Gross write-offs, 2021					0
Gross write-offs, 2020					118
Gross write-offs, 2019					3,036
Gross write-offs, prior					1,705
Six Months Ended June 30, 2023 Gross write-offs	2,292	$ 2,567	$ 360	$ 523	$ 4,859
Ratio Loan-To-Value					80.00%
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				$ 0
2022	0				0
2021	58,923				58,923
2020	202,884				202,884
2019	562,298				562,298
Prior	731,622				731,622
Residential whole loans, total or weighted average	1,555,727				1,555,727
Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	1,401				1,401
2020	12,770				12,770
2019	9,500				9,500
Prior	91,390				91,390
Residential whole loans, total or weighted average	115,061				115,061
Non-QM loans
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	46,910				46,910
2020	189,507				189,507
2019	433,303				433,303
Prior	243,106				243,106
Residential whole loans, total or weighted average	912,826				912,826
Gross write-offs, 2023					0
Gross write-offs, 2022					0
Gross write-offs, 2021					0
Gross write-offs, 2020					71
Gross write-offs, 2019					25
Gross write-offs, prior					110
Six Months Ended June 30, 2023 Gross write-offs					206
Non-QM loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	45,509				45,509
2020	176,737				176,737
2019	428,804				428,804
Prior	237,126				237,126
Residential whole loans, total or weighted average	888,176				888,176
Non-QM loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	1,401				1,401
2020	12,770				12,770
2019	4,499				4,499
Prior	5,980				5,980
Residential whole loans, total or weighted average	24,650				$ 24,650
Non-QM loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value					64.00%	65.00%
Transitional loans
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				$ 0
2022	0				0
2021	729				729
2020	3,915				3,915
2019	27,665				27,665
Prior	10,118				10,118
Residential whole loans, total or weighted average	42,427				42,427
Gross write-offs, 2023					0
Gross write-offs, 2022					0
Gross write-offs, 2021					0
Gross write-offs, 2020					47
Gross write-offs, 2019					2,560
Gross write-offs, prior					1,181
Six Months Ended June 30, 2023 Gross write-offs					3,788
Transitional loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	729				729
2020	3,915				3,915
2019	24,475				24,475
Prior	10,118				10,118
Residential whole loans, total or weighted average	39,237				39,237
Transitional loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	0				0
2020	0				0
2019	3,190				3,190
Prior	0				0
Residential whole loans, total or weighted average	3,190				$ 3,190
Transitional loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value					65.00%	66.00%
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				$ 0
2022	0				0
2021	12,685				12,685
2020	22,232				22,232
2019	110,830				110,830
Prior	46,033				46,033
Residential whole loans, total or weighted average	191,780				191,780
Gross write-offs, 2023					0
Gross write-offs, 2022					0
Gross write-offs, 2021					0
Gross write-offs, 2020					0
Gross write-offs, 2019					451
Gross write-offs, prior					0
Six Months Ended June 30, 2023 Gross write-offs					451
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	12,685				12,685
2020	22,232				22,232
2019	109,019				109,019
Prior	45,948				45,948
Residential whole loans, total or weighted average	189,884				189,884
Single-family rental loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	0				0
2020	0				0
2019	1,811				1,811
Prior	85				85
Residential whole loans, total or weighted average	1,896				$ 1,896
Single-family rental loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value					68.00%	69.00%
Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				$ 0
2022	0				0
2021	0				0
2020	0				0
2019	0				0
Prior	75,389				75,389
Residential whole loans, total or weighted average	75,389				75,389
Gross write-offs, 2023					0
Gross write-offs, 2022					0
Gross write-offs, 2021					0
Gross write-offs, 2020					0
Gross write-offs, 2019					0
Gross write-offs, prior					0
Six Months Ended June 30, 2023 Gross write-offs					0
Seasoned performing loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	0				0
2020	0				0
2019	0				0
Prior	72,915				72,915
Residential whole loans, total or weighted average	72,915				72,915
Seasoned performing loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	0				0
2020	0				0
2019	0				0
Prior	2,474				2,474
Residential whole loans, total or weighted average	2,474				$ 2,474
Seasoned performing loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value					29.00%	30.00%
Purchased Credit Deteriorated Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				$ 0
2022	0				0
2021	0				0
2020	0				0
2019	0				0
Prior	448,366				448,366
Residential whole loans, total or weighted average	448,366				448,366	$ 470,294
Gross write-offs, 2023					0
Gross write-offs, 2022					0
Gross write-offs, 2021					0
Gross write-offs, 2020					0
Gross write-offs, 2019					0
Gross write-offs, prior					414
Six Months Ended June 30, 2023 Gross write-offs					414
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	0				0
2020	0				0
2019	0				0
Prior	365,515				365,515
Residential whole loans, total or weighted average	365,515				365,515
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0				0
2022	0				0
2021	0				0
2020	0				0
2019	0				0
Prior	82,851				82,851
Residential whole loans, total or weighted average	$ 82,851				$ 82,851
Certain rehabilitation loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value					69.00%	70.00%