Interest Income and Interest Expense (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2021	Jun. 30, 2020	Jun. 30, 2021	Jun. 30, 2020
Interest Income Expense Net [Abstract]
Loans held for investment, net	$ 92,379	$ 79,164	$ 179,618	$ 174,284
Investment securities	31,539	41,527	61,491	89,664
Margin balances	6,286	6,364	12,377	16,495
Financial instruments owned	2,543	2,859	5,438	7,433
Other	844	(1,546)	2,207	1,669
Total interest income	133,591	128,368	261,131	289,545
Senior notes	11,971	13,073	24,074	24,266
Bank deposits	1,214	2,277	2,497	11,846
Federal Home Loan Bank advances	41	682	52	3,035
Other	952	(2,948)	1,995	(1,706)
Total interest expense	$ 14,178	$ 13,084	$ 28,618	$ 37,441