Residential Whole Loans (Additional Credit Related Information) (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Mar. 31, 2021	Mar. 31, 2020	Sep. 30, 2020	Dec. 31, 2020
Financing Receivable, Credit Quality Indicator [Line Items]
2021	$ 119,691
2020	546,721
2019	1,650,170
2018	751,683
2017	86,491
Prior	777,544
Residential whole loans, at carrying value, total or weighted average	3,932,300
Gross write-offs, 2020	0
Gross write-offs, 2019	991
Gross write-offs, 2018	12
Gross write-offs, prior	214
Three Months Ended March 31, 2021 Gross write-offs	1,217	$ 647
Net write-offs, 2020	0
Net write-offs, 2019	991
Net write-offs, 2018	12
Net write-offs	214
Total Recoveries	0
Three Months Ended March 31, 2021 Net write-offs	1,217
Purchased performing loans	$ 3,287,689			$ 3,521,624
Ratio Loan-To-Value	80.00%
Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	$ 115,420
2020	502,632
2019	1,617,755
2018	732,372
2017	80,192
Prior	551,316
Residential whole loans, at carrying value, total or weighted average	3,599,687
Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	4,271
2020	44,089
2019	32,415
2018	19,311
2017	6,299
Prior	226,228
Residential whole loans, at carrying value, total or weighted average	332,613
Non-QM Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	91,059
2020	463,139
2019	1,036,830
2018	579,713
2017	67,212
Prior	5,491
Residential whole loans, at carrying value, total or weighted average	2,243,444
Gross write-offs, 2019	0
Three Months Ended March 31, 2021 Gross write-offs	0
Net write-offs, 2019	0
Three Months Ended March 31, 2021 Net write-offs	0
Purchased performing loans	2,243,444			2,357,185
Non-QM Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	86,788
2020	419,564
2019	1,012,210
2018	560,489
2017	62,613
Prior	5,340
Residential whole loans, at carrying value, total or weighted average	2,147,004
Non-QM Loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	4,271
2020	43,575
2019	24,620
2018	19,224
2017	4,599
Prior	151
Residential whole loans, at carrying value, total or weighted average	96,440
Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	12,867
2020	43,504
2019	343,355
2018	58,888
2017	5,771
Prior	0
Residential whole loans, at carrying value, total or weighted average	464,385
Gross write-offs, 2020	0
Gross write-offs, 2019	991
Gross write-offs, 2018	12
Gross write-offs, prior	0
Three Months Ended March 31, 2021 Gross write-offs	1,003
Net write-offs, 2020	0
Net write-offs, 2019	991
Net write-offs, 2018	12
Net write-offs	0
Three Months Ended March 31, 2021 Net write-offs	1,003
Purchased performing loans	464,385			581,801
Rehabilitation loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	12,867
2020	43,504
2019	341,513
2018	58,888
2017	4,071
Prior	0
Residential whole loans, at carrying value, total or weighted average	460,843
Rehabilitation loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0
2020	0
2019	1,842
2018	0
2017	1,700
Prior	0
Residential whole loans, at carrying value, total or weighted average	3,542
Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	15,765
2020	40,078
2019	269,985
2018	113,082
2017	12,881
Prior	0
Residential whole loans, at carrying value, total or weighted average	451,791
Purchased performing loans	451,791			446,374
Single-family rental loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	15,765
2020	39,564
2019	264,032
2018	112,995
2017	12,881
Prior	0
Residential whole loans, at carrying value, total or weighted average	445,237
Single-family rental loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0
2020	514
2019	5,953
2018	87
2017	0
Prior	0
Residential whole loans, at carrying value, total or weighted average	6,554
Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0
2020	0
2019	0
2018	0
2017	0
Prior	128,069
Residential whole loans, at carrying value, total or weighted average	128,069
Purchased performing loans	128,069			136,264
Seasoned performing loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0
2020	0
2019	0
2018	0
2017	0
Prior	122,389
Residential whole loans, at carrying value, total or weighted average	122,389
Seasoned performing loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0
2020	0
2019	0
2018	0
2017	0
Prior	5,680
Residential whole loans, at carrying value, total or weighted average	5,680
Purchased Credit Deteriorated Loans
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0
2020	0
2019	0
2018	0
2017	627
Prior	643,984
Residential whole loans, at carrying value, total or weighted average	644,611
Gross write-offs, prior	214
Three Months Ended March 31, 2021 Gross write-offs	214
Net write-offs	214
Total Recoveries	0
Three Months Ended March 31, 2021 Net write-offs	214
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, Less than 80 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0
2020	0
2019	0
2018	0
2017	627
Prior	423,587
Residential whole loans, at carrying value, total or weighted average	424,214
Purchased Credit Deteriorated Loans | Debt-to-Value Ratio, 81 to 100 Percent
Financing Receivable, Credit Quality Indicator [Line Items]
2021	0
2020	0
2019	0
2018	0
2017	0
Prior	220,397
Residential whole loans, at carrying value, total or weighted average	220,397
Settled Whole Loans | Non-QM Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 2,228,899			2,336,117
Ratio Loan-To-Value	64.00%		64.00%
Settled Whole Loans | Rehabilitation loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 450,717			563,430
Ratio Loan-To-Value	64.00%		63.00%
Settled Whole Loans | Single-family rental loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 449,045			442,456
Ratio Loan-To-Value	70.00%		70.00%
Settled Whole Loans | Seasoned performing loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 128,003			136,157
Ratio Loan-To-Value	39.00%		40.00%
Settled Whole Loans | Certain Rehabilitation Loans
Financing Receivable, Credit Quality Indicator [Line Items]
Purchased performing loans	$ 151,700			$ 189,900
Ratio Loan-To-Value	68.00%		69.00%