Residential Whole Loans - Additional Credit Related Information (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2022	Sep. 30, 2022	Jun. 30, 2022	Mar. 31, 2022	Dec. 31, 2021	Sep. 30, 2021	Jun. 30, 2021	Mar. 31, 2021	Dec. 31, 2022	Dec. 31, 2021
Financing Receivable, Credit Quality Indicator [Line Items]
2022	$ 0								$ 0
2021	64,140								64,140
2020	234,089								234,089
2019	636,824								636,824
2018	300,737								300,737
Prior	590,739								590,739
Residential whole loans, total or weighted average	1,826,529								1,826,529
Gross write-offs, 2021									0
Gross write-offs, 2020									92
Gross write-offs, 2019									572
Gross write-offs, 2018									406
Gross write-offs, prior									400
Year Ended December 31, 2022 Gross write-offs	302	$ 282	$ 360	$ 523	$ 1,225	$ 533	$ 400	$ 1,217	$ 1,470
Weighted Average LTV Ratio									80.00%
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								$ 0
2021	62,012								62,012
2020	219,446								219,446
2019	624,349								624,349
2018	292,603								292,603
Prior	493,915								493,915
Residential whole loans, total or weighted average	1,692,325								1,692,325
Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	2,128								2,128
2020	14,643								14,643
2019	12,475								12,475
2018	8,134								8,134
Prior	96,824								96,824
Residential whole loans, total or weighted average	134,204								134,204
Non-QM Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	49,471								49,471
2020	203,587								203,587
2019	466,174								466,174
2018	238,478								238,478
Prior	29,572								29,572
Residential whole loans, total or weighted average	987,282								987,282
Gross write-offs, 2021									0
Gross write-offs, 2020									0
Gross write-offs, 2019									0
Gross write-offs, 2018									51
Gross write-offs, prior									0
Year Ended December 31, 2022 Gross write-offs									51
Non-QM Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	47,343								47,343
2020	189,257								189,257
2019	460,422								460,422
2018	232,400								232,400
Prior	28,516								28,516
Residential whole loans, total or weighted average	957,938								957,938
Non-QM Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	2,128								2,128
2020	14,330								14,330
2019	5,752								5,752
2018	6,078								6,078
Prior	1,056								1,056
Residential whole loans, total or weighted average	29,344								29,344
Transitional loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	1,182								1,182
2020	5,153								5,153
2019	49,409								49,409
2018	14,698								14,698
Prior	4,746								4,746
Residential whole loans, total or weighted average	75,188								75,188
Gross write-offs, 2021									0
Gross write-offs, 2020									92
Gross write-offs, 2019									367
Gross write-offs, 2018									287
Gross write-offs, prior									0
Year Ended December 31, 2022 Gross write-offs									746
Transitional loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	1,182								1,182
2020	5,153								5,153
2019	44,815								44,815
2018	12,727								12,727
Prior	3,047								3,047
Residential whole loans, total or weighted average	66,924								66,924
Transitional loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	0								0
2020	0								0
2019	4,594								4,594
2018	1,971								1,971
Prior	1,699								1,699
Residential whole loans, total or weighted average	8,264								8,264
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	13,487								13,487
2020	25,349								25,349
2019	121,241								121,241
2018	47,561								47,561
Prior	3,195								3,195
Residential whole loans, total or weighted average	210,833								210,833
Gross write-offs, 2021									0
Gross write-offs, 2020									0
Gross write-offs, 2019									205
Gross write-offs, 2018									68
Gross write-offs, prior									0
Year Ended December 31, 2022 Gross write-offs									273
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	13,487								13,487
2020	25,036								25,036
2019	119,112								119,112
2018	47,476								47,476
Prior	3,195								3,195
Residential whole loans, total or weighted average	208,306								208,306
Single-family rental loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	0								0
2020	313								313
2019	2,129								2,129
2018	85								85
Prior	0								0
Residential whole loans, total or weighted average	2,527								2,527
Seasoned Performing Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	0								0
2020	0								0
2019	0								0
2018	0								0
Prior	82,932								82,932
Residential whole loans, total or weighted average	82,932								82,932
Gross write-offs, 2021									0
Gross write-offs, 2020									0
Gross write-offs, 2019									0
Gross write-offs, 2018									0
Gross write-offs, prior									0
Year Ended December 31, 2022 Gross write-offs									0
Seasoned Performing Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	0								0
2020	0								0
2019	0								0
2018	0								0
Prior	80,341								80,341
Residential whole loans, total or weighted average	80,341								80,341
Seasoned Performing Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	0								0
2020	0								0
2019	0								0
2018	0								0
Prior	2,591								2,591
Residential whole loans, total or weighted average	2,591								2,591
Purchased Credit Deteriorated Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	0								0
2020	0								0
2019	0								0
2018	0								0
Prior	470,294								470,294
Residential whole loans, total or weighted average	470,294				547,772				470,294	$ 547,772
Gross write-offs, 2021									0
Gross write-offs, 2020									0
Gross write-offs, 2019									0
Gross write-offs, 2018									0
Gross write-offs, prior									400
Year Ended December 31, 2022 Gross write-offs									400
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	0								0
2020	0								0
2019	0								0
2018	0								0
Prior	378,816								378,816
Residential whole loans, total or weighted average	378,816								378,816
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2022	0								0
2021	0								0
2020	0								0
2019	0								0
2018	0								0
Prior	91,478								91,478
Residential whole loans, total or weighted average	91,478								91,478
One-to-four-family residential property loans | Transitional loans
Financing Receivable, Credit Quality Indicator [Line Items]
Securitization of eligible investor loans	$ 223,200				$ 137,300				$ 223,200	$ 137,300
Weighted Average LTV Ratio									70.00%	71.00%
Settled Whole Loans | Non-QM Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Weighted Average LTV Ratio									65.00%	66.00%
Settled Whole Loans | Transitional loans
Financing Receivable, Credit Quality Indicator [Line Items]
Weighted Average LTV Ratio									66.00%	67.00%
Settled Whole Loans | Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
Weighted Average LTV Ratio									69.00%	70.00%
Settled Whole Loans | Seasoned Performing Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Weighted Average LTV Ratio									30.00%	37.00%