Loans Held for Investment - Loan Products Type and Characteristics (Details) - USD ($) $ in Thousands	Jun. 30, 2018	Dec. 31, 2017
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Total Principal	$ 13,083,186	$ 13,525,820
Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 2,023,828	$ 2,182,383
Outstanding balance past due, percent of total balance (percent)	15.50%	16.10%
Bankruptcy | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 244,955	$ 272,778
Outstanding balance past due, percent of total balance (percent)	1.90%	2.00%
Foreclosure | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 262,100	$ 200,376
Outstanding balance past due, percent of total balance (percent)	2.00%	1.50%
REO | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 34,519	$ 31,008
Outstanding balance past due, percent of total balance (percent)	0.30%	0.20%
30 Days Delinquent | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 797,730	$ 888,535
Outstanding balance past due, percent of total balance (percent)	6.10%	6.60%
60 Days Delinquent | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 288,255	$ 353,810
Outstanding balance past due, percent of total balance (percent)	2.20%	2.60%
90 Days Delinquent | Residential Portfolio Segment
Securitized Loans Held For Investment By Consolidated Variable Interest Entities [Line Items]
Outstanding balance past due	$ 396,269	$ 435,876
Outstanding balance past due, percent of total balance (percent)	3.00%	3.20%