Residential Whole Loans - Additional Credit Related Information (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2021	Sep. 30, 2021	Jun. 30, 2021	Mar. 31, 2021	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2021	Dec. 31, 2020
Financing Receivable, Credit Quality Indicator [Line Items]
2021	$ 91,409								$ 91,409
2020	376,866								376,866
2019	1,018,053								1,018,053
2018	453,900								453,900
2017	53,571								53,571
Prior	653,300								653,300
Residential whole loans, total or weighted average	2,647,099								2,647,099
Gross write-offs, 2020									0
Gross write-offs, 2019									1,385
Gross write-offs									1,338
Gross write-offs, prior									527
Nine Months Ended September 30, 2021 Gross write-offs	1,225	$ 533	$ 400	$ 1,217	$ 830	$ 281	$ 627	$ 647	3,373
Net write-offs, 2020									0
Net write-offs, 2019									1,385
Net write-offs, 2018									1,338
Net write-offs, prior									527
Nine Months Ended September 30, 2021 Net write-offs									$ 3,373
Weighted Average LTV Ratio									80.00%
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	88,639								$ 88,639
2020	351,279								351,279
2019	1,000,821								1,000,821
2018	441,473								441,473
2017	49,985								49,985
Prior	506,845								506,845
Residential whole loans, total or weighted average	2,439,042								2,439,042
Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	2,770								2,770
2020	25,587								25,587
2019	17,232								17,232
2018	12,427								12,427
2017	3,586								3,586
Prior	146,455								146,455
Residential whole loans, total or weighted average	208,057								208,057
Non-QM Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	63,211								63,211
2020	315,911								315,911
2019	673,422								673,422
2018	353,795								353,795
2017	37,719								37,719
Prior	4,104								4,104
Residential whole loans, total or weighted average	1,448,162								1,448,162
Gross write-offs, 2020									0
Gross write-offs, 2019									0
Gross write-offs									37
Gross write-offs, prior									0
Nine Months Ended September 30, 2021 Gross write-offs									37
Net write-offs, 2020									0
Net write-offs, 2019									0
Net write-offs, 2018									37
Net write-offs, prior									0
Nine Months Ended September 30, 2021 Net write-offs									37
Non-QM Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	60,441								60,441
2020	290,836								290,836
2019	662,258								662,258
2018	342,250								342,250
2017	35,833								35,833
Prior	3,954								3,954
Residential whole loans, total or weighted average	1,395,572								1,395,572
Non-QM Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	2,770								2,770
2020	25,075								25,075
2019	11,164								11,164
2018	11,545								11,545
2017	1,886								1,886
Prior	150								150
Residential whole loans, total or weighted average	52,590								52,590
Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	12,754								12,754
2020	24,716								24,716
2019	152,388								152,388
2018	22,330								22,330
2017	5,127								5,127
Prior	0								0
Residential whole loans, total or weighted average	217,315								217,315
Gross write-offs, 2020									0
Gross write-offs, 2019									1,329
Gross write-offs									1,296
Gross write-offs, prior									0
Nine Months Ended September 30, 2021 Gross write-offs									2,748
Net write-offs, 2020									0
Net write-offs, 2019									1,329
Net write-offs, 2018									1,296
Net write-offs, prior									0
Nine Months Ended September 30, 2021 Net write-offs									2,748
Rehabilitation loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	12,754								12,754
2020	24,716								24,716
2019	151,632								151,632
2018	21,534								21,534
2017	3,427								3,427
Prior	0								0
Residential whole loans, total or weighted average	214,063								214,063
Rehabilitation loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0								0
2020	0								0
2019	756								756
2018	796								796
2017	1,700								1,700
Prior	0								0
Residential whole loans, total or weighted average	3,252								3,252
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	15,444								15,444
2020	36,239								36,239
2019	192,243								192,243
2018	77,775								77,775
2017	10,107								10,107
Prior	0								0
Residential whole loans, total or weighted average	331,808								331,808
Gross write-offs, 2020									0
Gross write-offs, 2019									56
Gross write-offs									5
Gross write-offs, prior									0
Nine Months Ended September 30, 2021 Gross write-offs									61
Net write-offs, 2020									0
Net write-offs, 2019									56
Net write-offs, 2018									5
Net write-offs, prior									0
Nine Months Ended September 30, 2021 Net write-offs									61
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	15,444								15,444
2020	35,727								35,727
2019	186,931								186,931
2018	77,689								77,689
2017	10,107								10,107
Prior	0								0
Residential whole loans, total or weighted average	325,898								325,898
Single-family rental loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0								0
2020	512								512
2019	5,312								5,312
2018	86								86
2017	0								0
Prior	0								0
Residential whole loans, total or weighted average	5,910								5,910
Seasoned Performing Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	0								0
Prior	102,042								102,042
Residential whole loans, total or weighted average	102,042								102,042
Gross write-offs, 2020									0
Gross write-offs, 2019									0
Gross write-offs									0
Gross write-offs, prior									0
Nine Months Ended September 30, 2021 Gross write-offs									0
Net write-offs, 2020									0
Net write-offs, 2019									0
Net write-offs, 2018									0
Net write-offs, prior									0
Nine Months Ended September 30, 2021 Net write-offs									0
Seasoned Performing Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	0								0
Prior	98,288								98,288
Residential whole loans, total or weighted average	98,288								98,288
Seasoned Performing Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	0								0
Prior	3,754								3,754
Residential whole loans, total or weighted average	3,754								3,754
Purchased Credit Deteriorated Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	618								618
Prior	547,154								547,154
Residential whole loans, total or weighted average	547,772				673,708				547,772	$ 673,708
Gross write-offs, 2020									0
Gross write-offs, 2019									0
Gross write-offs									0
Gross write-offs, prior									527
Nine Months Ended September 30, 2021 Gross write-offs									527
Net write-offs, 2020									0
Net write-offs, 2019									0
Net write-offs, 2018									0
Net write-offs, prior									527
Nine Months Ended September 30, 2021 Net write-offs									527
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	618								618
Prior	404,603								404,603
Residential whole loans, total or weighted average	405,221								405,221
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0								0
2020	0								0
2019	0								0
2018	0								0
2017	0								0
Prior	142,551								142,551
Residential whole loans, total or weighted average	142,551								142,551
Settled Whole Loans | Non-QM Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	3,453,242				2,336,117				$ 3,453,242	$ 2,336,117
Weighted Average LTV Ratio									66.00%	64.00%
Settled Whole Loans | Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	727,964				563,430				$ 727,964	$ 563,430
Weighted Average LTV Ratio									67.00%	63.00%
Settled Whole Loans | Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	949,772				442,456				$ 949,772	$ 442,456
Weighted Average LTV Ratio									70.00%	70.00%
Settled Whole Loans | Seasoned Performing Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	101,995				136,157				$ 101,995	$ 136,157
Weighted Average LTV Ratio									37.00%	40.00%
Settled Whole Loans | Certain rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 137,300				$ 189,900				$ 137,300	$ 189,900
Weighted Average LTV Ratio									71.00%	69.00%