Residential Whole Loans - Allowance for Credit Losses (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2021	Sep. 30, 2021	Jun. 30, 2021	Mar. 31, 2021	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2021	Jan. 01, 2020
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	$ 44,102	$ 54,261	$ 63,244	$ 86,833	$ 106,246	$ 136,589	$ 218,011	$ 3,025	$ 86,833
Transition adjustment on adoption of ASU 2016-13								70,555	70,600
Current provision	(3,430)	(9,626)	(8,583)	(22,372)	(18,583)	(30,062)	(10,614)	74,897
Write-offs	(1,225)	(533)	(400)	(1,217)	(830)	(281)	(627)	(647)	(3,373)
Valuation adjustment on loans held for sale							(70,181)	70,181
Ending balance	39,447	44,102	54,261	63,244	86,833	106,246	136,589	218,011	39,447
Loans	2,647,099								2,647,099
Allowance for credit loss	39,447	44,102	54,261	63,244	86,833	106,246	136,589	218,011	39,447
Gross up of the amortized cost basis of Purchased Credit Deteriorated Loans									62,400
2021	91,409								91,409
2020	376,866								376,866
2019	1,018,053								1,018,053
2018	453,900								453,900
2017	53,571								53,571
Prior	653,300								653,300
Recoveries									0
Net write-offs									3,373
Gross write-offs, prior									527
Net write-offs, prior									527
Gross write-offs, 2017									123
Net write-offs, 2017									123
Gross write-offs, 2018									1,338
Net write-offs, 2018									1,338
Gross write-offs, 2019									1,385
Net write-offs, 2019									1,385
Gross write-offs, 2020									0
Net write-offs, 2020									0
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	2,439,042								2,439,042
2021	88,639								88,639
2020	351,279								351,279
2019	1,000,821								1,000,821
2018	441,473								441,473
2017	49,985								49,985
Prior	506,845								506,845
Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	208,057								208,057
2021	2,770								2,770
2020	25,587								25,587
2019	17,232								17,232
2018	12,427								12,427
2017	3,586								3,586
Prior	146,455								146,455
Non-QM Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	9,689	12,092	14,545	21,068	26,753	31,353	103,831	388	21,068
Transition adjustment on adoption of ASU 2016-13								6,904
Current provision	(1,400)	(2,403)	(2,416)	(6,523)	(5,599)	(4,568)	(2,297)	26,358
Write-offs	0	0	(37)	0	(86)	(32)	0	0
Valuation adjustment on loans held for sale							(70,181)	70,181
Ending balance	8,289	9,689	12,092	14,545	21,068	26,753	31,353	103,831	8,289
Allowance for credit loss	8,289	9,689	12,092	14,545	21,068	26,753	31,353	103,831	8,289
Rehabilitation loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	8,685	11,604	13,668	18,371	22,633	30,000	35,633	2,331	18,371
Transition adjustment on adoption of ASU 2016-13								517
Current provision	(706)	(2,526)	(1,809)	(3,700)	(3,837)	(7,140)	(5,213)	33,213
Write-offs	(1,098)	(393)	(255)	(1,003)	(425)	(227)	(420)	(428)
Valuation adjustment on loans held for sale							0	0
Ending balance	6,881	8,685	11,604	13,668	18,371	22,633	30,000	35,633	6,881
Loans	87,000				143,400				87,000
Allowance for credit loss	6,881	8,685	11,604	13,668	18,371	22,633	30,000	35,633	6,881
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	1,634	2,360	2,746	3,918	5,025	6,931	7,431	62	3,918
Transition adjustment on adoption of ASU 2016-13								754
Current provision	(178)	(670)	(386)	(1,172)	(1,107)	(1,906)	(500)	6,615
Write-offs	(5)	(56)	0	0	0	0	0	0
Valuation adjustment on loans held for sale							0	0
Ending balance	1,451	1,634	2,360	2,746	3,918	5,025	6,931	7,431	1,451
Allowance for credit loss	1,451	1,634	2,360	2,746	3,918	5,025	6,931	7,431	1,451
Seasoned Performing Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	50	57	66	107	150	224	249	0	107
Transition adjustment on adoption of ASU 2016-13								19
Current provision	(4)	(7)	(9)	(41)	(43)	(74)	(25)	230
Write-offs	0	0	0	0	0	0	0	0
Valuation adjustment on loans held for sale							0	0
Ending balance	46	50	57	66	107	150	224	249	46
Allowance for credit loss	46	50	57	66	107	150	224	249	46
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance	24,044	28,148	32,219	43,369	51,685	68,081	70,867	244	43,369
Transition adjustment on adoption of ASU 2016-13								62,361
Current provision	(1,142)	(4,020)	(3,963)	(10,936)	(7,997)	(16,374)	(2,579)	8,481
Write-offs	(122)	(84)	(108)	(214)	(319)	(22)	(207)	(219)
Valuation adjustment on loans held for sale							0	0
Ending balance	22,780	24,044	28,148	32,219	43,369	51,685	68,081	70,867	22,780
Loans	57,400				72,700				57,400
Allowance for credit loss	22,780	$ 24,044	$ 28,148	32,219	43,369	$ 51,685	$ 68,081	$ 70,867	22,780
Unfunded Loan Commitment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning balance				$ 1,200					1,200
Ending balance	205				1,200				205
Commitment to lend, unfunded	18,500				73,200				18,500
Allowance for credit loss	205				1,200				205
Non-QM Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Write-offs									(37)
Loans	1,448,162								1,448,162
2021	63,211								63,211
2020	315,911								315,911
2019	673,422								673,422
2018	353,795								353,795
2017	37,719								37,719
Prior	4,104								4,104
Recoveries									0
Net write-offs									37
Gross write-offs, prior									0
Net write-offs, prior									0
Gross write-offs, 2017									0
Net write-offs, 2017									0
Gross write-offs, 2018									37
Net write-offs, 2018									37
Gross write-offs, 2019									0
Net write-offs, 2019									0
Gross write-offs, 2020									0
Net write-offs, 2020									0
Non-QM Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	1,395,572								1,395,572
2021	60,441								60,441
2020	290,836								290,836
2019	662,258								662,258
2018	342,250								342,250
2017	35,833								35,833
Prior	3,954								3,954
Non-QM Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	52,590								52,590
2021	2,770								2,770
2020	25,075								25,075
2019	11,164								11,164
2018	11,545								11,545
2017	1,886								1,886
Prior	150								150
Rehabilitation loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Write-offs									(2,748)
Loans	217,315								217,315
2021	12,754								12,754
2020	24,716								24,716
2019	152,388								152,388
2018	22,330								22,330
2017	5,127								5,127
Prior	0								0
Recoveries									0
Net write-offs									2,748
Gross write-offs, prior									0
Net write-offs, prior									0
Gross write-offs, 2017									123
Net write-offs, 2017									123
Gross write-offs, 2018									1,296
Net write-offs, 2018									1,296
Gross write-offs, 2019									1,329
Net write-offs, 2019									1,329
Gross write-offs, 2020									0
Net write-offs, 2020									0
Rehabilitation loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	214,063								214,063
2021	12,754								12,754
2020	24,716								24,716
2019	151,632								151,632
2018	21,534								21,534
2017	3,427								3,427
Prior	0								0
Rehabilitation loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	3,252								3,252
2021	0								0
2020	0								0
2019	756								756
2018	796								796
2017	1,700								1,700
Prior	0								0
Single-family rental loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Write-offs									(61)
Loans	331,808								331,808
2021	15,444								15,444
2020	36,239								36,239
2019	192,243								192,243
2018	77,775								77,775
2017	10,107								10,107
Prior	0								0
Recoveries									0
Net write-offs									61
Gross write-offs, prior									0
Net write-offs, prior									0
Gross write-offs, 2017									0
Net write-offs, 2017									0
Gross write-offs, 2018									5
Net write-offs, 2018									5
Gross write-offs, 2019									56
Net write-offs, 2019									56
Gross write-offs, 2020									0
Net write-offs, 2020									0
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	325,898								325,898
2021	15,444								15,444
2020	35,727								35,727
2019	186,931								186,931
2018	77,689								77,689
2017	10,107								10,107
Prior	0								0
Single-family rental loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	5,910								5,910
2021	0								0
2020	512								512
2019	5,312								5,312
2018	86								86
2017	0								0
Prior	0								0
Seasoned Performing Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Write-offs									0
Loans	102,042								102,042
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	0								0
Prior	102,042								102,042
Recoveries									0
Net write-offs									0
Gross write-offs, prior									0
Net write-offs, prior									0
Gross write-offs, 2017									0
Net write-offs, 2017									0
Gross write-offs, 2018									0
Net write-offs, 2018									0
Gross write-offs, 2019									0
Net write-offs, 2019									0
Gross write-offs, 2020									0
Net write-offs, 2020									0
Seasoned Performing Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	98,288								98,288
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	0								0
Prior	98,288								98,288
Seasoned Performing Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	3,754								3,754
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	0								0
Prior	3,754								3,754
Purchased Credit Deteriorated Loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Write-offs									(527)
Loans	547,772				$ 673,708				547,772
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	618								618
Prior	547,154								547,154
Recoveries									0
Net write-offs									527
Gross write-offs, prior									527
Net write-offs, prior									527
Gross write-offs, 2017									0
Net write-offs, 2017									0
Gross write-offs, 2018									0
Net write-offs, 2018									0
Gross write-offs, 2019									0
Net write-offs, 2019									0
Gross write-offs, 2020									0
Net write-offs, 2020									0
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	405,221								405,221
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	618								618
Prior	404,603								404,603
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Loans	142,551								142,551
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	0								0
Prior	$ 142,551								$ 142,551
Cumulative Effect, Period of Adoption, Adjustment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Allowance for credit loss										$ 8,300