Residential Whole Loans (Additional Credit Related Information) (Details) - USD ($) $ in Thousands	3 Months Ended						9 Months Ended	12 Months Ended
	Sep. 30, 2021	Jun. 30, 2021	Mar. 31, 2021	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Sep. 30, 2021	Dec. 31, 2020
Financing Receivable, Credit Quality Indicator [Line Items]
2021	$ 105,270						$ 105,270
2020	415,756						415,756
2019	1,215,425						1,215,425
2018	546,523						546,523
2017	59,843						59,843
Prior	689,149						689,149
Residential whole loans, total or weighted average	3,031,966						3,031,966
Gross write-offs, 2020							0
Gross write-offs, 2019							1,115
Gross write-offs, 2018							505
Gross write-offs, 2017							123
Gross write-offs, prior							406
Nine Months Ended September 30, 2021 Gross write-offs	533	$ 400	$ 1,217	$ 281	$ 627	$ 647	2,149
Net write-offs, 2020							0
Net write-offs, 2019							1,115
Net write-offs, 2018							505
Net write-offs, 2017							123
Net write-offs							406
Nine Months Ended September 30, 2021 Net write-offs							$ 2,149
Ratio Loan-To-Value							80.00%
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	101,079						$ 101,079
2020	386,874						386,874
2019	1,193,084						1,193,084
2018	530,954						530,954
2017	54,966						54,966
Prior	530,840						530,840
Residential whole loans, total or weighted average	2,797,797						2,797,797
Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	4,191						4,191
2020	28,882						28,882
2019	22,341						22,341
2018	15,569						15,569
2017	4,877						4,877
Prior	158,309						158,309
Residential whole loans, total or weighted average	234,169						234,169
Non-QM loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	76,051						76,051
2020	350,244						350,244
2019	786,165						786,165
2018	422,629						422,629
2017	43,558						43,558
Prior	4,378						4,378
Residential whole loans, total or weighted average	1,683,025						1,683,025
Gross write-offs, 2018							37
Nine Months Ended September 30, 2021 Gross write-offs							37
Net write-offs, 2018							37
Nine Months Ended September 30, 2021 Net write-offs							37
Non-QM loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	71,860						71,860
2020	321,875						321,875
2019	770,456						770,456
2018	407,943						407,943
2017	40,381						40,381
Prior	4,228						4,228
Residential whole loans, total or weighted average	1,616,743						1,616,743
Non-QM loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	4,191						4,191
2020	28,369						28,369
2019	15,709						15,709
2018	14,686						14,686
2017	3,177						3,177
Prior	150						150
Residential whole loans, total or weighted average	66,282						66,282
Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	13,717						13,717
2020	27,910						27,910
2019	215,844						215,844
2018	32,024						32,024
2017	5,127						5,127
Prior	0						0
Residential whole loans, total or weighted average	294,622						294,622
Gross write-offs, 2019							1,059
Gross write-offs, 2018							468
Gross write-offs, 2017							123
Nine Months Ended September 30, 2021 Gross write-offs							1,650
Net write-offs, 2019							1,059
Net write-offs, 2018							468
Net write-offs, 2017							123
Nine Months Ended September 30, 2021 Net write-offs							1,650
Rehabilitation loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	13,717						13,717
2020	27,910						27,910
2019	214,618						214,618
2018	31,228						31,228
2017	3,427						3,427
Prior	0						0
Residential whole loans, total or weighted average	290,900						290,900
Rehabilitation loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0						0
2020	0						0
2019	1,226						1,226
2018	796						796
2017	1,700						1,700
Prior	0						0
Residential whole loans, total or weighted average	3,722						3,722
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	15,502						15,502
2020	37,602						37,602
2019	213,416						213,416
2018	91,870						91,870
2017	10,537						10,537
Prior	0						0
Residential whole loans, total or weighted average	368,927						368,927
Gross write-offs, 2019							56
Nine Months Ended September 30, 2021 Gross write-offs							56
Net write-offs, 2019							56
Nine Months Ended September 30, 2021 Net write-offs							56
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	15,502						15,502
2020	37,089						37,089
2019	208,010						208,010
2018	91,783						91,783
2017	10,537						10,537
Prior	0						0
Residential whole loans, total or weighted average	362,921						362,921
Single-family rental loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0						0
2020	513						513
2019	5,406						5,406
2018	87						87
2017	0						0
Prior	0						0
Residential whole loans, total or weighted average	6,006						6,006
Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0						0
2020	0						0
2019	0						0
2018	0						0
2017	0						0
Prior	110,162						110,162
Residential whole loans, total or weighted average	110,162						110,162
Seasoned performing loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0						0
2020	0						0
2019	0						0
2018	0						0
2017	0						0
Prior	105,348						105,348
Residential whole loans, total or weighted average	105,348						105,348
Seasoned performing loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0						0
2020	0						0
2019	0						0
2018	0						0
2017	0						0
Prior	4,814						4,814
Residential whole loans, total or weighted average	4,814						4,814
Purchased credit deteriorated loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0						0
2020	0						0
2019	0						0
2018	0						0
2017	621						621
Prior	574,609						574,609
Residential whole loans, total or weighted average	575,230						575,230	$ 673,708
Gross write-offs, prior							406
Nine Months Ended September 30, 2021 Gross write-offs							406
Net write-offs							406
Nine Months Ended September 30, 2021 Net write-offs							406
Purchased credit deteriorated loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0						0
2020	0						0
2019	0						0
2018	0						0
2017	621						621
Prior	421,264						421,264
Residential whole loans, total or weighted average	421,885						421,885
Purchased credit deteriorated loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0						0
2020	0						0
2019	0						0
2018	0						0
2017	0						0
Prior	153,345						153,345
Residential whole loans, total or weighted average	153,345						153,345
Settled Whole Loans | Non-QM loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	2,825,883						$ 2,825,883	$ 2,336,117
Ratio Loan-To-Value							64.00%	64.00%
Settled Whole Loans | Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	587,539						$ 587,539	$ 563,430
Ratio Loan-To-Value							66.00%	63.00%
Settled Whole Loans | Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	739,428						$ 739,428	$ 442,456
Ratio Loan-To-Value							70.00%	70.00%
Settled Whole Loans | Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	110,112						$ 110,112	$ 136,157
Ratio Loan-To-Value							38.00%	40.00%
Settled Whole Loans | Certain rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 142,700						$ 142,700	$ 189,900
Ratio Loan-To-Value			69.00%				70.00%