Residential Whole Loans - Additional Credit Related Information (Details) - USD ($) $ in Thousands	3 Months Ended		12 Months Ended
	Mar. 31, 2023	Mar. 31, 2022	Dec. 31, 2022
Financing Receivable, Credit Quality Indicator [Line Items]
2023	$ 0
2022	0
2021	63,199
2020	226,136
2019	603,828
Prior	859,916
Residential whole loans, total or weighted average	1,753,080
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	1,598
Gross write-offs, prior	968
Three Months Ended March 31, 2023 Gross write-offs	$ 2,567	$ 523
Ratio Loan-To-Value	80.00%
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	$ 0
2022	0
2021	61,079
2020	213,320
2019	592,776
Prior	764,170
Residential whole loans, total or weighted average	1,631,345
Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	2,120
2020	12,816
2019	11,053
Prior	95,746
Residential whole loans, total or weighted average	121,734
Non-QM loans
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	49,025
2020	197,861
2019	453,299
Prior	257,914
Residential whole loans, total or weighted average	958,099
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	0
Gross write-offs, prior	0
Three Months Ended March 31, 2023 Gross write-offs	0
Non-QM loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	46,905
2020	185,045
2019	447,565
Prior	251,532
Residential whole loans, total or weighted average	931,047
Non-QM loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	2,120
2020	12,816
2019	5,734
Prior	6,382
Residential whole loans, total or weighted average	$ 27,052
Non-QM loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value	65.00%		65.00%
Transitional loans
Financing Receivable, Credit Quality Indicator [Line Items]
2023	$ 0
2022	0
2021	729
2020	4,738
2019	35,308
Prior	12,497
Residential whole loans, total or weighted average	53,272
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	1,148
Gross write-offs, prior	855
Three Months Ended March 31, 2023 Gross write-offs	2,003
Transitional loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	729
2020	4,738
2019	32,118
Prior	12,497
Residential whole loans, total or weighted average	50,082
Transitional loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	0
2020	0
2019	3,190
Prior	0
Residential whole loans, total or weighted average	$ 3,190
Transitional loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value	65.00%		66.00%
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2023	$ 0
2022	0
2021	13,445
2020	23,537
2019	115,221
Prior	49,360
Residential whole loans, total or weighted average	201,563
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	451
Gross write-offs, prior	0
Three Months Ended March 31, 2023 Gross write-offs	451
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	13,445
2020	23,537
2019	113,092
Prior	49,275
Residential whole loans, total or weighted average	199,350
Single-family rental loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	0
2020	0
2019	2,128
Prior	85
Residential whole loans, total or weighted average	$ 2,213
Single-family rental loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value	69.00%		69.00%
Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
2023	$ 0
2022	0
2021	0
2020	0
2019	0
Prior	79,465
Residential whole loans, total or weighted average	79,465
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	0
Gross write-offs, prior	0
Three Months Ended March 31, 2023 Gross write-offs	0
Seasoned performing loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	0
2020	0
2019	0
Prior	76,938
Residential whole loans, total or weighted average	76,938
Seasoned performing loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	0
2020	0
2019	0
Prior	2,527
Residential whole loans, total or weighted average	$ 2,527
Seasoned performing loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value	30.00%		30.00%
Purchased Credit Deteriorated Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2023	$ 0
2022	0
2021	0
2020	0
2019	0
Prior	460,680
Residential whole loans, total or weighted average	460,680		$ 470,294
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	0
Gross write-offs, prior	113
Three Months Ended March 31, 2023 Gross write-offs	113
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	0
2020	0
2019	0
Prior	373,929
Residential whole loans, total or weighted average	373,929
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2023	0
2022	0
2021	0
2020	0
2019	0
Prior	86,752
Residential whole loans, total or weighted average	$ 86,752
Certain rehabilitation loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio Loan-To-Value	69.00%		70.00%