Residential Whole Loans (Additional Credit Related Information) (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
2020	$ 569,835								$ 569,835
2019	1,880,880								1,880,880
2018	833,693								833,693
2017	95,833								95,833
2016	11,960								11,960
Prior	803,131								803,131
Total	4,195,332								4,195,332
Gross write-offs, 2019									21
Gross write-offs, 2018									1,564
Gross write-offs, 2017									32
Gross write-offs, prior									768
Year Ended December 31, 2020 Gross write-offs	830	$ 281	$ 627	$ 647	$ 277	$ 62	$ 50	$ 0	2,385
Net write-offs, 2019									21
Net write-offs, 2018									1,564
Net write-offs, 2017									32
Net write-offs, prior									768
Total Recoveries									0
Year Ended December 31, 2020 Net write-offs									2,385
Purchased performing loans	3,521,624				5,370,653				3,521,624	$ 5,370,653
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	507,736								507,736
2019	1,827,345								1,827,345
2018	808,770								808,770
2017	85,499								85,499
2016	10,469								10,469
Prior	557,509								557,509
Total	3,797,328								3,797,328
Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	62,099								62,099
2019	53,535								53,535
2018	24,923								24,923
2017	10,334								10,334
2016	1,491								1,491
Prior	245,622								245,622
Total	398,004								398,004
Non-QM Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	489,172								489,172
2019	1,140,719								1,140,719
2018	645,364								645,364
2017	76,181								76,181
2016	5,749								5,749
Prior	0								0
Total	2,357,185								2,357,185
Gross write-offs, 2018									117
Year Ended December 31, 2020 Gross write-offs									117
Net write-offs, 2018									117
Total Recoveries									0
Year Ended December 31, 2020 Net write-offs									117
Purchased performing loans	2,357,185				3,707,245				2,357,185	3,707,245
Non-QM Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	429,241								429,241
2019	1,111,534								1,111,534
2018	621,201								621,201
2017	67,547								67,547
2016	5,597								5,597
Prior	0								0
Total	2,235,120								2,235,120
Non-QM Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	59,931								59,931
2019	29,185								29,185
2018	24,163								24,163
2017	8,634								8,634
2016	152								152
Prior	0								0
Total	122,065								122,065
Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	44,927								44,927
2019	460,377								460,377
2018	70,594								70,594
2017	5,903								5,903
2016	0								0
Prior	0								0
Total	581,801								581,801
Gross write-offs, 2019									21
Gross write-offs, 2018									1,447
Gross write-offs, 2017									32
Year Ended December 31, 2020 Gross write-offs									1,500
Net write-offs, 2019									21
Net write-offs, 2018									1,447
Net write-offs, 2017									32
Total Recoveries									0
Year Ended December 31, 2020 Net write-offs									1,500
Purchased performing loans	581,801				1,026,097				581,801	1,026,097
Rehabilitation loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	44,153								44,153
2019	448,646								448,646
2018	70,046								70,046
2017	4,203								4,203
2016	0								0
Prior	0								0
Total	567,048								567,048
Rehabilitation loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	774								774
2019	11,731								11,731
2018	548								548
2017	1,700								1,700
2016	0								0
Prior	0								0
Total	14,753								14,753
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	35,736								35,736
2019	279,784								279,784
2018	117,735								117,735
2017	13,119								13,119
2016	0								0
Prior	0								0
Total	446,374								446,374
Purchased performing loans	446,374				460,742				446,374	460,742
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	34,342								34,342
2019	267,165								267,165
2018	117,523								117,523
2017	13,119								13,119
2016	0								0
Prior	0								0
Total	432,149								432,149
Single-family rental loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	1,394								1,394
2019	12,619								12,619
2018	212								212
2017	0								0
2016	0								0
Prior	0								0
Total	14,225								14,225
Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0								0
2019	0								0
2018	0								0
2017	0								0
2016	79								79
Prior	136,185								136,185
Total	136,264								136,264
Purchased performing loans	136,264				176,569				136,264	176,569
Seasoned performing loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0								0
2019	0								0
2018	0								0
2017	0								0
2016	0								0
Prior	130,316								130,316
Total	130,316								130,316
Seasoned performing loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0								0
2019	0								0
2018	0								0
2017	0								0
2016	79								79
Prior	5,869								5,869
Total	5,948								5,948
Purchased Credit Deteriorated Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0								0
2019	0								0
2018	0								0
2017	630								630
2016	6,132								6,132
Prior	666,946								666,946
Total	673,708								673,708
Gross write-offs, prior									768
Year Ended December 31, 2020 Gross write-offs									768
Net write-offs, prior									768
Total Recoveries									0
Year Ended December 31, 2020 Net write-offs									768
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0								0
2019	0								0
2018	0								0
2017	630								630
2016	4,872								4,872
Prior	427,193								427,193
Total	432,695								432,695
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0								0
2019	0								0
2018	0								0
2017	0								0
2016	1,260								1,260
Prior	239,753								239,753
Total	241,013								241,013
Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total	4,108,499				6,066,345				4,108,499	6,066,345
Settled Whole Loans | Non-QM Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	2,336,117				3,706,857				$ 2,336,117	$ 3,706,857
Weighted average LTV ratio									64.00%	67.00%
Settled Whole Loans | Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	563,430				1,023,766				$ 563,430	$ 1,023,766
Weighted average LTV ratio									63.00%	64.00%
Settled Whole Loans | Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	442,456				460,679				$ 442,456	$ 460,679
Weighted average LTV ratio									70.00%	70.00%
Settled Whole Loans | Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	136,157				176,569				$ 136,157	$ 176,569
Weighted average LTV ratio									40.00%	46.00%
Settled Whole Loans | Certain rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 189,900				$ 269,200				$ 189,900	$ 269,200
Weighted average LTV ratio									68.00%	69.00%