Residential Whole Loans - Additional Credit Related Information (Details) - USD ($) $ in Thousands	3 Months Ended		12 Months Ended
	Mar. 31, 2024	Mar. 31, 2023	Dec. 31, 2023
Financing Receivable, Credit Quality Indicator [Line Items]
2024	$ 0
2023	0
2022	0
2021	51,752
2020	197,803
Prior	1,234,815
Total	1,484,370
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	(59)
Gross write-offs, prior	438
Gross write-offs	$ 379	$ 2,567
Ratio loan- to-value (as a percent)	80.00%
Debt-to-Value Ratio, Less Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	$ 0
2023	0
2022	0
2021	50,366
2020	187,226
Prior	1,153,943
Total	1,391,535
Debt-to-Value Ratio, Greater Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	1,386
2020	10,577
Prior	80,872
Total	92,835
Non-QM loans
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	43,735
2020	174,832
Prior	598,050
Total	816,617
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	0
Gross write-offs, prior	0
Gross write-offs	$ 0
Non-QM loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio loan- to-value (as a percent)	65.00%		65.00%
Non-QM loans | Debt-to-Value Ratio, Less Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	$ 0
2023	0
2022	0
2021	42,349
2020	164,663
Prior	591,730
Total	798,742
Non-QM loans | Debt-to-Value Ratio, Greater Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	1,386
2020	10,169
Prior	6,320
Total	17,875
Transitional loans
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	156
2020	3,915
Prior	25,027
Total	29,098
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	0
Gross write-offs, prior	416
Gross write-offs	$ 416
Transitional loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio loan- to-value (as a percent)	64.00%		64.00%
Transitional loans | Debt-to-Value Ratio, Less Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	$ 0
2023	0
2022	0
2021	156
2020	3,915
Prior	25,027
Total	29,098
Transitional loans | Debt-to-Value Ratio, Greater Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	0
2020	0
Prior	0
Total	0
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	7,861
2020	19,056
Prior	122,026
Total	148,943
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	(59)
Gross write-offs, prior	0
Gross write-offs	(59)
Single-family rental loans | Debt-to-Value Ratio, Less Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	7,861
2020	18,648
Prior	109,439
Total	135,948
Single-family rental loans | Debt-to-Value Ratio, Greater Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	0
2020	408
Prior	12,587
Total	12,995
Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	0
2020	0
Prior	66,065
Total	66,065
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	0
Gross write-offs, prior	0
Gross write-offs	$ 0
Seasoned performing loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio loan- to-value (as a percent)	28.00%		28.00%
Seasoned performing loans | Debt-to-Value Ratio, Less Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	$ 0
2023	0
2022	0
2021	0
2020	0
Prior	63,739
Total	63,739
Seasoned performing loans | Debt-to-Value Ratio, Greater Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	0
2020	0
Prior	2,326
Total	2,326
Purchased credit deteriorated loans
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	0
2020	0
Prior	423,647
Total	423,647		$ 429,726
Gross write-offs, 2023	0
Gross write-offs, 2022	0
Gross write-offs, 2021	0
Gross write-offs, 2020	0
Gross write-offs, 2019	0
Gross write-offs, prior	22
Gross write-offs	22
Purchased credit deteriorated loans | Debt-to-Value Ratio, Less Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	0
2020	0
Prior	364,008
Total	364,008
Purchased credit deteriorated loans | Debt-to-Value Ratio, Greater Than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2024	0
2023	0
2022	0
2021	0
2020	0
Prior	59,639
Total	59,639
Certain rehabilitation loans | Transitional Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 608,900		$ 551,300
Certain rehabilitation loans | Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Ratio loan- to-value (as a percent)	67.00%		68.00%