Residential Whole Loans (Interest Income Components) (Details) - Residential whole loans - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2021	Sep. 30, 2020	Sep. 30, 2021	Sep. 30, 2020
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 79,602	$ 70,948	$ 213,156	$ 261,819
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	10,504	8,784	30,097	27,265
Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	48,394	45,609	124,327	180,041
Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	23,891	25,884	68,047	112,212
Performing | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	9,918	10,863	23,914	39,502
Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	9,497	6,917	23,484	21,528
Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,728	1,945	5,260	6,799
Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	3,360	0	3,622	0
Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	20,704	16,555	58,732	54,513
Carrying Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	41,551	54,393	134,169	207,306
Carrying Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	10,504	8,784	30,097	27,265
Carrying Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	31,047	45,609	104,072	180,041
Carrying Value | Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	17,437	25,884	59,789	112,212
Carrying Value | Performing | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	5,808	10,863	19,429	39,502
Carrying Value | Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	6,074	6,917	19,594	21,528
Carrying Value | Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	1,728	1,945	5,260	6,799
Carrying Value | Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Carrying Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	38,051	16,555	78,987	54,513
Fair Value | Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Fair Value | Performing
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	17,347	0	20,255	0
Fair Value | Performing | Non-QM loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	6,454	0	8,258	0
Fair Value | Performing | Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	4,110	0	4,485	0
Fair Value | Performing | Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	3,423	0	3,890	0
Fair Value | Performing | Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	0	0	0	0
Fair Value | Performing | Agency eligible investor loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	3,360	0	3,622	0
Fair Value | Nonperforming
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest income	$ 20,704	$ 16,555	$ 58,732	$ 54,513