Interest Income and Interest Expense (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Interest Income Expense Net [Abstract]
Loans held for investment, net									$ 331,813	$ 379,848	$ 300,541
Investment securities									149,915	231,021	252,200
Margin balances									28,155	52,008	49,515
Financial instruments owned									12,594	23,528	21,407
Other									1,355	38,477	22,786
Total interest income									523,832	724,882	646,449
Senior notes									54,063	44,507	44,610
Bank deposits									14,550	95,813	82,256
Federal Home Loan Bank advances									3,667	7,872	15,173
Other									(6,502)	29,739	28,037
Total interest expense	$ 14,515	$ 13,822	$ 13,084	$ 24,357	$ 31,448	$ 44,144	$ 52,891	$ 49,448	$ 65,778	$ 177,931	$ 170,076