Residential Whole Loans - Additional Credit Related Information (Details) - USD ($) $ in Thousands	3 Months Ended				6 Months Ended	12 Months Ended
	Jun. 30, 2022	Mar. 31, 2022	Jun. 30, 2021	Mar. 31, 2021	Jun. 30, 2022	Dec. 31, 2021
Financing Receivable, Credit Quality Indicator [Line Items]
2022	$ 0				$ 0
2021	69,861				69,861
2020	272,065				272,065
2019	744,102				744,102
2018	343,510				343,510
Prior	633,677				633,677
Residential whole loans, total or weighted average	2,063,215				2,063,215
Gross write-offs, 2019					495
Gross write-offs, 2018					105
Gross write-offs, prior					284
Six Months Ended June 30, 2022 Gross write-offs	360	$ 523	$ 400	$ 1,217	$ 884
Ratio Loan-To-Value					80.00%
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				$ 0
2021	67,714				67,714
2020	253,692				253,692
2019	719,573				719,573
2018	332,011				332,011
Prior	522,309				522,309
Residential whole loans, total or weighted average	1,895,299				1,895,299
Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	2,147				2,147
2020	18,373				18,373
2019	24,529				24,529
2018	11,499				11,499
Prior	111,368				111,368
Residential whole loans, total or weighted average	167,916				167,916
Non-QM loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	52,635				52,635
2020	228,966				228,966
2019	527,819				527,819
2018	272,137				272,137
Prior	32,139				32,139
Residential whole loans, total or weighted average	1,113,696				1,113,696
Gross write-offs, 2018					51
Six Months Ended June 30, 2022 Gross write-offs					51
Non-QM loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	50,488				50,488
2020	213,384				213,384
2019	519,616				519,616
2018	264,276				264,276
Prior	30,517				30,517
Residential whole loans, total or weighted average	1,078,281				1,078,281
Non-QM loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	2,147				2,147
2020	15,582				15,582
2019	8,203				8,203
2018	7,861				7,861
Prior	1,622				1,622
Residential whole loans, total or weighted average	35,415				$ 35,415
Non-QM loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value					65.00%	66.00%
Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				$ 0
2021	3,061				3,061
2020	14,103				14,103
2019	71,662				71,662
2018	17,592				17,592
Prior	4,746				4,746
Residential whole loans, total or weighted average	111,164				111,164
Gross write-offs, 2019					296
Gross write-offs, 2018					41
Six Months Ended June 30, 2022 Gross write-offs					337
Rehabilitation loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	3,061				3,061
2020	11,823				11,823
2019	60,678				60,678
2018	14,040				14,040
Prior	3,047				3,047
Residential whole loans, total or weighted average	92,649				92,649
Rehabilitation loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	0				0
2020	2,280				2,280
2019	10,984				10,984
2018	3,552				3,552
Prior	1,699				1,699
Residential whole loans, total or weighted average	18,515				$ 18,515
Rehabilitation loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value					67.00%	67.00%
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				$ 0
2021	14,165				14,165
2020	28,996				28,996
2019	144,621				144,621
2018	53,781				53,781
Prior	5,566				5,566
Residential whole loans, total or weighted average	247,129				247,129
Gross write-offs, 2019					199
Gross write-offs, 2018					13
Six Months Ended June 30, 2022 Gross write-offs					212
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	14,165				14,165
2020	28,485				28,485
2019	139,279				139,279
2018	53,695				53,695
Prior	5,566				5,566
Residential whole loans, total or weighted average	241,190				241,190
Single-family rental loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	0				0
2020	511				511
2019	5,342				5,342
2018	86				86
Prior	0				0
Residential whole loans, total or weighted average	5,939				$ 5,939
Single-family rental loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value					70.00%	70.00%
Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				$ 0
2021	0				0
2020	0				0
2019	0				0
2018	0				0
Prior	92,909				92,909
Residential whole loans, total or weighted average	92,909				92,909
Seasoned performing loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	0				0
2020	0				0
2019	0				0
2018	0				0
Prior	89,973				89,973
Residential whole loans, total or weighted average	89,973				89,973
Seasoned performing loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	0				0
2020	0				0
2019	0				0
2018	0				0
Prior	2,936				2,936
Residential whole loans, total or weighted average	2,936				$ 2,936
Seasoned performing loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value					31.00%	37.00%
Purchased Credit Deteriorated Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				$ 0
2021	0				0
2020	0				0
2019	0				0
2018	0				0
Prior	498,317				498,317
Residential whole loans, total or weighted average	498,317				498,317	$ 547,772
Gross write-offs, prior					284
Six Months Ended June 30, 2022 Gross write-offs					284
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	0				0
2020	0				0
2019	0				0
2018	0				0
Prior	393,206				393,206
Residential whole loans, total or weighted average	393,206				393,206
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0				0
2021	0				0
2020	0				0
2019	0				0
2018	0				0
Prior	105,111				105,111
Residential whole loans, total or weighted average	105,111				105,111
Certain rehabilitation loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 172,300				$ 172,300	$ 137,300
Ratio Loan-To-Value					72.00%	71.00%