Loans Held for Investment - Loan Products Type and Characteristics (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Unpaid Principal Balance	$ 13,284,786	$ 13,169,843
Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 2,295,052	$ 2,379,192
% of Unpaid Principal Balance	17.30%	18.10%
30 Days Delinquent | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 967,534	$ 1,070,173
% of Unpaid Principal Balance	7.30%	8.10%
60 Days Delinquent | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 333,270	$ 336,950
% of Unpaid Principal Balance	2.50%	2.60%
90 Days Delinquent | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 339,452	$ 291,866
% of Unpaid Principal Balance	2.60%	2.20%
Bankruptcy | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 299,129	$ 322,288
% of Unpaid Principal Balance	2.30%	2.40%
Foreclosure | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 296,935	$ 298,526
% of Unpaid Principal Balance	2.20%	2.30%
Loss Mitigation | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 4	$ 0
% of Unpaid Principal Balance	0.00%	0.00%
REO | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 58,728	$ 59,389
% of Unpaid Principal Balance	0.40%	0.50%