Residential Whole Loans - Additional Credit Related Information (Details) - USD ($) $ in Thousands	3 Months Ended		12 Months Ended
	Mar. 31, 2022	Mar. 31, 2021	Dec. 31, 2021
Financing Receivable, Credit Quality Indicator [Line Items]
2022	$ 0
2021	76,777
2020	317,399
2019	868,882
2018	390,020
Prior	666,970
Residential whole loans, total or weighted average	2,320,048
Gross write-offs, 2019	226
Gross write-offs, 2018	71
Gross write-offs, prior	226
Three Months Ended March 31, 2022 Gross write-offs	$ 523	$ 1,217
Ratio Loan-To-Value	80.00%
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	$ 0
2021	74,015
2020	295,689
2019	842,329
2018	374,784
Prior	530,216
Residential whole loans, total or weighted average	2,117,033
Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	2,762
2020	21,710
2019	26,553
2018	15,236
Prior	136,754
Residential whole loans, total or weighted average	203,015
Non-QM loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	57,984
2020	265,020
2019	593,873
2018	312,961
Prior	35,893
Residential whole loans, total or weighted average	1,265,731
Gross write-offs, 2018	51
Three Months Ended March 31, 2022 Gross write-offs	51
Non-QM loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	55,222
2020	246,102
2019	584,293
2018	302,545
Prior	34,265
Residential whole loans, total or weighted average	1,222,427
Non-QM loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	2,762
2020	18,918
2019	9,580
2018	10,416
Prior	1,628
Residential whole loans, total or weighted average	43,304
Non-QM loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 3,621,124		$ 3,453,242
Ratio Loan-To-Value	65.00%		66.00%
Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	$ 0
2021	4,046
2020	21,622
2019	104,677
2018	19,407
Prior	4,756
Residential whole loans, total or weighted average	154,508
Gross write-offs, 2019	199
Gross write-offs, 2018	20
Three Months Ended March 31, 2022 Gross write-offs	219
Rehabilitation loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	4,046
2020	19,342
2019	92,938
2018	14,673
Prior	3,057
Residential whole loans, total or weighted average	134,056
Rehabilitation loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	0
2020	2,280
2019	11,739
2018	4,734
Prior	1,699
Residential whole loans, total or weighted average	20,452
Rehabilitation loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 885,155		$ 727,964
Ratio Loan-To-Value	67.00%		67.00%
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	$ 0
2021	14,747
2020	30,757
2019	170,332
2018	57,652
Prior	9,602
Residential whole loans, total or weighted average	283,090
Gross write-offs, 2019	27
Three Months Ended March 31, 2022 Gross write-offs	27
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	14,747
2020	30,245
2019	165,098
2018	57,566
Prior	9,602
Residential whole loans, total or weighted average	277,258
Single-family rental loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	0
2020	512
2019	5,234
2018	86
Prior	0
Residential whole loans, total or weighted average	5,832
Single-family rental loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 1,152,195		$ 949,772
Ratio Loan-To-Value	70.00%		70.00%
Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	$ 0
2021	0
2020	0
2019	0
2018	0
Prior	98,269
Residential whole loans, total or weighted average	98,269
Seasoned performing loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	0
2020	0
2019	0
2018	0
Prior	95,307
Residential whole loans, total or weighted average	95,307
Seasoned performing loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	0
2020	0
2019	0
2018	0
Prior	2,962
Residential whole loans, total or weighted average	2,962
Seasoned performing loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 98,224		$ 101,995
Ratio Loan-To-Value	36.00%		37.00%
Purchased Credit Deteriorated Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	$ 0
2021	0
2020	0
2019	0
2018	0
Prior	518,450
Residential whole loans, total or weighted average	518,450		$ 547,772
Gross write-offs, prior	226
Three Months Ended March 31, 2022 Gross write-offs	226
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	0
2020	0
2019	0
2018	0
Prior	387,985
Residential whole loans, total or weighted average	387,985
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0
2021	0
2020	0
2019	0
2018	0
Prior	130,465
Residential whole loans, total or weighted average	130,465
Certain rehabilitation loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 160,500		$ 137,300
Ratio Loan-To-Value	74.00%		71.00%