Securitizations and Variable Interest Entities - Summary of Select Information Related to VIEs (Details) - USD ($) $ in Thousands		Dec. 31, 2019	Dec. 31, 2018
Variable Interest Entity [Line Items]
Restricted cash	[1]	$ 243,343	$ 271,084
Securities available for sale at fair value		270,927	170,469
Loans held for investment, Fair value	[1]	1,079,315	1,883,251
Loans held for investment by the Company at fair value	[1]	43,693	2,583
Loans held for sale by the Company at fair value	[1]	722,355	840,021
Accrued interest receivable	[1]	12,857	22,255
Other assets	[1]	143,668	124,967
Total assets		2,982,341	3,819,527
Accrued interest payable	[1]	9,260	19,241
Accrued expenses and other liabilities	[1]	142,636	152,118
Notes, certificates and secured borrowings, Fair Value	[1]	1,081,466	1,905,875
Credit facilities and securities sold under repurchase agreements	[1]	587,453	458,802
Payable to securitization note and certificate holders	[1]	40,610	256,354
Total liabilities		2,082,154	2,948,546
Consolidated VIEs
Variable Interest Entity [Line Items]
Restricted cash		30,046	43,918
Securities available for sale at fair value		0	0
Loans held for investment, Fair value		197,842	642,094
Loans held for investment by the Company at fair value		40,251	0
Loans held for sale by the Company at fair value		551,455	739,216
Accrued interest receivable		4,431	10,438
Other assets		1,359	2,498
Total assets		825,384	1,438,164
Accrued interest payable		3,185	7,594
Accrued expenses and other liabilities		244	1,627
Notes, certificates and secured borrowings, Fair Value		197,842	648,908
Credit facilities and securities sold under repurchase agreements		387,251	306,790
Payable to securitization note and certificate holders		40,610	256,354
Total liabilities		629,132	1,221,273
Net Assets		196,252	216,891
Unconsolidated VIEs
Variable Interest Entity [Line Items]
Restricted cash		0	0
Securities available for sale at fair value		220,135	116,768
Loans held for investment, Fair value		0	0
Loans held for investment by the Company at fair value		0
Loans held for sale by the Company at fair value		0	0
Accrued interest receivable		877	1,214
Other assets		52,098	29,206
Total assets		273,110	147,188
Accrued interest payable		0	0
Accrued expenses and other liabilities		0	0
Notes, certificates and secured borrowings, Fair Value		0	0
Credit facilities and securities sold under repurchase agreements		0	57,012
Payable to securitization note and certificate holders		0	0
Total liabilities		0	57,012
Net Assets		273,110	90,176
Total
Variable Interest Entity [Line Items]
Restricted cash		30,046	43,918
Securities available for sale at fair value		220,135	116,768
Loans held for investment, Fair value		197,842	642,094
Loans held for investment by the Company at fair value		40,251
Loans held for sale by the Company at fair value		551,455	739,216
Accrued interest receivable		5,308	11,652
Other assets		53,457	31,704
Total assets		1,098,494	1,585,352
Accrued interest payable		3,185	7,594
Accrued expenses and other liabilities		244	1,627
Notes, certificates and secured borrowings, Fair Value		197,842	648,908
Credit facilities and securities sold under repurchase agreements		387,251	363,802
Payable to securitization note and certificate holders		40,610	256,354
Total liabilities		629,132	1,278,285
Net Assets		$ 469,362	$ 307,067

[1] Includes amounts in consolidated variable interest entities (VIEs) presented separately in the table below.