Residential Whole Loans (Additional Credit Related Information) (Details) - USD ($) $ in Thousands	3 Months Ended						9 Months Ended
	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Sep. 30, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
2020	$ 490,345						$ 490,345
2019	2,091,705						2,091,705
2018	948,938						948,938
2017	107,208						107,208
2016	12,768						12,768
Prior	842,842						842,842
Residential whole loans, at carrying value, total or weighted average	4,493,806						4,493,806
Gross write-offs, 2019							13
Gross write-offs, 2018							1,062
Gross write-offs, 2017							32
Gross write-offs, prior							448
Nine Months Ended September 30, 2020 Gross write-offs	281	$ 627	$ 647	$ 62	$ 50	$ 0	1,555
Net write-offs, 2019							13
Net write-offs, 2018							1,062
Net write-offs, 2017							32
Net write-offs							448
Total Recoveries							0
Nine Months Ended September 30, 2020 Net write-offs							1,555
Purchased Performing Loans	3,791,792						$ 3,791,792	$ 5,370,653
Ratio Loan-To-Value							80.00%
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	439,607						$ 439,607
2019	2,026,153						2,026,153
2018	918,394						918,394
2017	96,397						96,397
2016	9,432						9,432
Prior	559,803						559,803
Residential whole loans, at carrying value, total or weighted average	4,049,786						4,049,786
Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	50,738						50,738
2019	65,552						65,552
2018	30,544						30,544
2017	10,811						10,811
2016	3,336						3,336
Prior	283,039						283,039
Residential whole loans, at carrying value, total or weighted average	444,020						444,020
Non-QM Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	428,811						428,811
2019	1,231,653						1,231,653
2018	714,188						714,188
2017	83,973						83,973
2016	6,523						6,523
Prior	0						0
Residential whole loans, at carrying value, total or weighted average	2,465,148						2,465,148
Gross write-offs, 2018							32
Nine Months Ended September 30, 2020 Gross write-offs							32
Net write-offs, 2018							32
Nine Months Ended September 30, 2020 Net write-offs							32
Purchased Performing Loans	2,465,148						2,465,148	3,707,245
Non-QM Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	380,729						380,729
2019	1,196,185						1,196,185
2018	684,404						684,404
2017	74,862						74,862
2016	6,371						6,371
Prior	0						0
Residential whole loans, at carrying value, total or weighted average	2,342,551						2,342,551
Non-QM Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	48,082						48,082
2019	35,468						35,468
2018	29,784						29,784
2017	9,111						9,111
2016	152						152
Prior	0						0
Residential whole loans, at carrying value, total or weighted average	122,597						122,597
Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	37,740						37,740
2019	558,361						558,361
2018	94,521						94,521
2017	9,246						9,246
2016	0						0
Prior	0						0
Residential whole loans, at carrying value, total or weighted average	699,868						699,868
Gross write-offs, 2019							13
Gross write-offs, 2018							1,030
Gross write-offs, 2017							32
Gross write-offs, prior							0
Nine Months Ended September 30, 2020 Gross write-offs							1,075
Net write-offs, 2019							13
Net write-offs, 2018							1,030
Net write-offs, 2017							32
Net write-offs							0
Nine Months Ended September 30, 2020 Net write-offs							1,075
Purchased Performing Loans	699,868						699,868	1,026,097
Rehabilitation loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	36,478						36,478
2019	542,865						542,865
2018	93,973						93,973
2017	7,546						7,546
2016	0						0
Prior	0						0
Residential whole loans, at carrying value, total or weighted average	680,862						680,862
Rehabilitation loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	1,262						1,262
2019	15,496						15,496
2018	548						548
2017	1,700						1,700
2016	0						0
Prior	0						0
Residential whole loans, at carrying value, total or weighted average	19,006						19,006
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	23,794						23,794
2019	301,691						301,691
2018	140,229						140,229
2017	13,356						13,356
2016	0						0
Prior	0						0
Residential whole loans, at carrying value, total or weighted average	479,070						479,070
Purchased Performing Loans	479,070						479,070	460,742
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	22,400						22,400
2019	287,103						287,103
2018	140,017						140,017
2017	13,356						13,356
2016	0						0
Prior	0						0
Residential whole loans, at carrying value, total or weighted average	462,876						462,876
Single-family rental loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	1,394						1,394
2019	14,588						14,588
2018	212						212
2017	0						0
2016	0						0
Prior	0						0
Residential whole loans, at carrying value, total or weighted average	16,194						16,194
Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0						0
2019	0						0
2018	0						0
2017	0						0
2016	79						79
Prior	147,627						147,627
Residential whole loans, at carrying value, total or weighted average	147,706						147,706
Purchased Performing Loans	147,706						147,706	176,569
Seasoned performing loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0						0
2019	0						0
2018	0						0
2017	0						0
2016	79						79
Prior	139,538						139,538
Residential whole loans, at carrying value, total or weighted average	139,617						139,617
Seasoned performing loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0						0
2019	0						0
2018	0						0
2017	0						0
2016	0						0
Prior	8,089						8,089
Residential whole loans, at carrying value, total or weighted average	8,089						8,089
Purchased Credit Deteriorated Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0						0
2019	0						0
2018	0						0
2017	633						633
2016	6,166						6,166
Prior	695,215						695,215
Residential whole loans, at carrying value, total or weighted average	702,014						702,014
Gross write-offs, prior							448
Nine Months Ended September 30, 2020 Gross write-offs							448
Net write-offs							448
Total Recoveries							0
Nine Months Ended September 30, 2020 Net write-offs							448
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0						0
2019	0						0
2018	0						0
2017	633						633
2016	2,982						2,982
Prior	420,265						420,265
Residential whole loans, at carrying value, total or weighted average	423,880						423,880
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, 80 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2020	0						0
2019	0						0
2018	0						0
2017	0						0
2016	3,184						3,184
Prior	274,950						274,950
Residential whole loans, at carrying value, total or weighted average	278,134						278,134
Settled Whole Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Residential whole loans, at carrying value, total or weighted average	4,387,559						4,387,559	6,066,345
Settled Whole Loans | Non-QM Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased Performing Loans	2,438,395						$ 2,438,395	3,706,857
Ratio Loan-To-Value			67.00%				64.00%
Settled Whole Loans | Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased Performing Loans	677,235						$ 677,235	1,023,766
Ratio Loan-To-Value			64.00%				63.00%
Settled Whole Loans | Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased Performing Loans	474,045						$ 474,045	460,679
Ratio Loan-To-Value			70.00%				70.00%
Settled Whole Loans | Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased Performing Loans	147,556						$ 147,556	176,569
Ratio Loan-To-Value			46.00%				41.00%
Settled Whole Loans | Certain Rehabilitation Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased Performing Loans	$ 222,200						$ 222,200	$ 269,200
Ratio Loan-To-Value			69.00%				68.00%