Securitizations and Variable Interest Entities - Summary of Select Information Related to VIEs (Details) - USD ($) $ in Thousands		Jun. 30, 2020	Dec. 31, 2019
Variable Interest Entity [Line Items]
Restricted cash	[1]	$ 134,345	$ 243,343
Securities available for sale at fair value		221,930	270,927
Loans held for investment at fair value	[1]	785,228	1,079,315
Loans held for investment by the Company at fair value	[1]	65,557	43,693
Loans held for sale by the Company at fair value	[1]	587,093	722,355
Accrued interest receivable	[1]	11,314	12,857
Other assets	[1]	109,702	143,668
Total assets		2,452,599	2,982,341
Accrued interest payable	[1]	7,780	9,260
Accrued expenses and other liabilities	[1]	86,369	142,636
Notes, certificates and secured borrowings at fair value	[1]	785,928	1,081,466
Payable to Structured Program note and certificate holders at fair value	[1]	193,034	40,610
Credit facilities and securities sold under repurchase agreements	[1]	480,079	587,453
Liabilities		1,706,457	2,082,154
Total
Variable Interest Entity [Line Items]
Restricted cash		38,465	30,046
Securities available for sale at fair value		190,206	220,135
Loans held for investment at fair value		101,742	197,842
Loans held for investment by the Company at fair value		64,634	40,251
Loans held for sale by the Company at fair value		482,726	551,455
Accrued interest receivable		5,017	5,308
Other assets		44,114	53,457
Total assets		926,904	1,098,494
Accrued interest payable		2,032	3,185
Accrued expenses and other liabilities		103	244
Notes, certificates and secured borrowings at fair value		101,742	197,842
Payable to Structured Program note and certificate holders at fair value		193,034	40,610
Credit facilities and securities sold under repurchase agreements		294,510	387,251
Liabilities		591,421	629,132
Total net assets		335,483	469,362
Unconsolidated VIEs
Variable Interest Entity [Line Items]
Restricted cash		0	0
Securities available for sale at fair value		190,206	220,135
Loans held for investment at fair value		0	0
Loans held for investment by the Company at fair value		0	0
Loans held for sale by the Company at fair value		0	0
Accrued interest receivable		99	877
Other assets		43,319	52,098
Total assets		233,624	273,110
Accrued interest payable		0	0
Accrued expenses and other liabilities		0	0
Notes, certificates and secured borrowings at fair value		0	0
Payable to Structured Program note and certificate holders at fair value		0	0
Credit facilities and securities sold under repurchase agreements		0	0
Liabilities		0	0
Total net assets		233,624	273,110
Consolidated VIEs
Variable Interest Entity [Line Items]
Restricted cash		38,465	30,046
Securities available for sale at fair value		0	0
Loans held for investment at fair value		101,742	197,842
Loans held for investment by the Company at fair value		64,634	40,251
Loans held for sale by the Company at fair value		482,726	551,455
Accrued interest receivable		4,918	4,431
Other assets		795	1,359
Total assets		693,280	825,384
Accrued interest payable		2,032	3,185
Accrued expenses and other liabilities		103	244
Notes, certificates and secured borrowings at fair value		101,742	197,842
Payable to Structured Program note and certificate holders at fair value		193,034	40,610
Credit facilities and securities sold under repurchase agreements		294,510	387,251
Liabilities		591,421	629,132
Total net assets		$ 101,859	$ 196,252

[1] Includes amounts in consolidated variable interest entities (VIEs) presented separately in the table below.