Residential Whole Loans (LTV on Loans) (Details) - 90 Days or More Past Due: $ in Thousands	Mar. 31, 2021 USD ($)	Dec. 31, 2020 USD ($)
Purchased Credit Deteriorated Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 117,509	$ 119,621
UPB	$ 142,850	$ 145,028
LTV	0.859	0.867
Non-QM Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 135,260	$ 148,387
UPB	$ 132,732	$ 144,681
LTV	0.657	0.659
Rehabilitation loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 136,991	$ 136,347
UPB	$ 136,991	$ 136,347
LTV	0.657	0.658
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 24,091	$ 20,388
UPB	$ 24,052	$ 20,233
LTV	0.734	0.727
Seasoned performing loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 8,721	$ 8,031
UPB	$ 9,449	$ 8,823
LTV	0.507	0.551
Residential Loans At Fair Value
Accounts, Notes, Loans and Financing Receivable [Line Items]
Carrying Value / Fair Value	$ 555,171	$ 571,729
UPB	$ 584,025	$ 625,621
LTV	0.826	0.868