Residential Whole Loans - Additional Credit Related Information (Details) - USD ($) $ in Thousands	3 Months Ended						9 Months Ended	12 Months Ended
	Sep. 30, 2022	Jun. 30, 2022	Mar. 31, 2022	Sep. 30, 2021	Jun. 30, 2021	Mar. 31, 2021	Sep. 30, 2022	Dec. 31, 2021
Financing Receivable, Credit Quality Indicator [Line Items]
2022	$ 0						$ 0
2021	65,903						65,903
2020	248,562						248,562
2019	675,984						675,984
2018	319,020						319,020
Prior	605,808						605,808
Residential whole loans, total or weighted average	1,915,277						1,915,277
Gross write-offs, 2020							92
Gross write-offs, 2019							522
Gross write-offs, 2018							160
Gross write-offs, prior							391
Nine Months Ended September 30, 2022 Gross write-offs	282	$ 360	$ 523	$ 533	$ 400	$ 1,217	$ 1,165
Ratio Loan-To-Value							80.00%
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						$ 0
2021	63,768						63,768
2020	233,777						233,777
2019	661,427						661,427
2018	308,592						308,592
Prior	503,327						503,327
Residential whole loans, total or weighted average	1,770,891						1,770,891
Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	2,135						2,135
2020	14,785						14,785
2019	14,557						14,557
2018	10,428						10,428
Prior	102,481						102,481
Residential whole loans, total or weighted average	144,386						144,386
Non-QM loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	50,044						50,044
2020	213,889						213,889
2019	483,733						483,733
2018	252,698						252,698
Prior	30,142						30,142
Residential whole loans, total or weighted average	1,030,506						1,030,506
Gross write-offs, 2018							51
Nine Months Ended September 30, 2022 Gross write-offs							51
Non-QM loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	47,909						47,909
2020	199,510						199,510
2019	476,591						476,591
2018	245,908						245,908
Prior	29,082						29,082
Residential whole loans, total or weighted average	999,000						999,000
Non-QM loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	2,135						2,135
2020	14,379						14,379
2019	7,142						7,142
2018	6,790						6,790
Prior	1,060						1,060
Residential whole loans, total or weighted average	31,506						$ 31,506
Non-QM loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value							65.00%	66.00%
Residential transition loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						$ 0
2021	2,037						2,037
2020	7,698						7,698
2019	61,664						61,664
2018	16,617						16,617
Prior	4,746						4,746
Residential whole loans, total or weighted average	92,762						92,762
Gross write-offs, 2020							92
Gross write-offs, 2019							317
Gross write-offs, 2018							41
Nine Months Ended September 30, 2022 Gross write-offs							450
Residential transition loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	2,037						2,037
2020	7,698						7,698
2019	57,070						57,070
2018	13,065						13,065
Prior	3,047						3,047
Residential whole loans, total or weighted average	82,917						82,917
Residential transition loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	0						0
2020	0						0
2019	4,594						4,594
2018	3,552						3,552
Prior	1,699						1,699
Residential whole loans, total or weighted average	9,845						$ 9,845
Residential transition loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value							66.00%	67.00%
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						$ 0
2021	13,822						13,822
2020	26,975						26,975
2019	130,587						130,587
2018	49,705						49,705
Prior	3,213						3,213
Residential whole loans, total or weighted average	224,302						224,302
Gross write-offs, 2019							205
Gross write-offs, 2018							68
Nine Months Ended September 30, 2022 Gross write-offs							273
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	13,822						13,822
2020	26,569						26,569
2019	127,766						127,766
2018	49,619						49,619
Prior	3,213						3,213
Residential whole loans, total or weighted average	220,989						220,989
Single-family rental loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	0						0
2020	406						406
2019	2,821						2,821
2018	86						86
Prior	0						0
Residential whole loans, total or weighted average	3,313						$ 3,313
Single-family rental loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value							69.00%	70.00%
Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						$ 0
2021	0						0
2020	0						0
2019	0						0
2018	0						0
Prior	87,028						87,028
Residential whole loans, total or weighted average	87,028						87,028
Seasoned performing loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	0						0
2020	0						0
2019	0						0
2018	0						0
Prior	84,120						84,120
Residential whole loans, total or weighted average	84,120						84,120
Seasoned performing loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	0						0
2020	0						0
2019	0						0
2018	0						0
Prior	2,908						2,908
Residential whole loans, total or weighted average	2,908						$ 2,908
Seasoned performing loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value							30.00%	37.00%
Purchased Credit Deteriorated Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						$ 0
2021	0						0
2020	0						0
2019	0						0
2018	0						0
Prior	480,679						480,679
Residential whole loans, total or weighted average	480,679						480,679	$ 547,772
Gross write-offs, prior							391
Nine Months Ended September 30, 2022 Gross write-offs							391
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	0						0
2020	0						0
2019	0						0
2018	0						0
Prior	383,865						383,865
Residential whole loans, total or weighted average	383,865						383,865
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0						0
2021	0						0
2020	0						0
2019	0						0
2018	0						0
Prior	96,814						96,814
Residential whole loans, total or weighted average	$ 96,814						$ 96,814
Certain rehabilitation loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value							71.00%	71.00%