Residential Whole Loans (Residential Whole Loans) (Details) $ in Thousands		Jun. 30, 2021 USD ($) loan	Dec. 31, 2020 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		$ 3,470,385
Allowance for Credit Losses		(54,261)	$ (86,833)
Total Residential Whole Loans	[1],[2]	$ 5,550,979	$ 5,325,401
Number of loans | loan		18,592	18,734
Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses		$ (54,261)	$ (86,833)
Total Residential Whole Loans		$ 3,416,124	$ 4,108,499
Number of loans | loan		11,434	13,112
Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses		$ 0	$ 0
Total Residential Whole Loans		$ 2,134,855	$ 1,216,902
Number of loans | loan		7,158	5,622
Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		$ 3,827,632	$ 3,521,624
Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		2,861,228	3,521,624
Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		966,404	0
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		1,168,451	1,216,902
Nonperforming | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		0	0
Nonperforming | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		1,168,451	1,216,902
Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		1,960,855
Non-QM loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		2,440,714	2,357,185
Non-QM loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		1,960,855	2,357,185
Non-QM loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		479,859	0
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		364,463
Rehabilitation loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		432,697	581,801
Rehabilitation loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		364,463	581,801
Rehabilitation loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		68,234	0
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		417,544
Single-family rental loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		527,229	446,374
Single-family rental loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		417,544	446,374
Single-family rental loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		109,685	0
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		118,366
Seasoned performing loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		118,366	136,264
Seasoned performing loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		118,366	136,264
Seasoned performing loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		0	0
Agency eligible investor loans | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		308,626	0
Agency eligible investor loans | Performing | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		0	0
Agency eligible investor loans | Performing | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		308,626	0
Purchased credit deteriorated loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		609,157	673,708
Purchased credit deteriorated loans | Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		609,157	673,708
Purchased credit deteriorated loans | Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans		$ 0	$ 0

[1] Includes approximately $2.3 billion and $1.8 billion of Residential whole loans transferred to consolidated variable interest entities (“VIEs”) at June 30, 2021 and December 31, 2020, respectively. Such assets can be used only to settle the obligations of each respective VIE.

[2] See Note 6 for information regarding the Company’s pledged assets.