Fair Value of Financial Instruments - Schedule of Fair Value Hierarchies for Financial Assets and Liabilities Measured on a Recurring Basis (Details) - USD ($) $ in Thousands	Mar. 31, 2021	Dec. 31, 2020
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	$ 110,458	$ 90,732
Equity securities	145,022	123,838
Asset derivatives	12,688	11,529
Total other investments, at fair value	218,471	219,701
Mortgage servicing rights (included in other assets)	20,894	14,758
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	1,095	2,090
Securities sold, not yet purchased	39,625	46,637
Other equity securities
Assets, Fair Value Disclosure [Abstract]
Equity securities	33,010	28,885
Exchange Traded Funds
Assets, Fair Value Disclosure [Abstract]
Equity securities	64,293	63,875
Invesque Inc.
Assets, Fair Value Disclosure [Abstract]
Equity securities	47,719	31,078
Significant unobservable inputs Level 3
Assets, Fair Value Disclosure [Abstract]
Total assets	29,178	23,965
Liabilities, Fair Value Disclosure [Abstract]
Contingent consideration payable	200	200
Recurring | Quoted prices in active markets Level 1
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	1,356	1,355
Loans, at fair value	0	0
Asset derivatives	667	2,090
Total other investments, at fair value	667	2,090
Mortgage servicing rights (included in other assets)	0	0
Total assets	147,010	127,248
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	0	0
Securities sold, not yet purchased	27,052	16,479
Contingent consideration payable	0	0
Total liabilities	27,052	16,479
Recurring | Quoted prices in active markets Level 1 | U.S. Treasury securities and obligations of U.S. government authorities and agencies
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	0	0
Recurring | Quoted prices in active markets Level 1 | Obligations of state and political subdivisions
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	0	0
Recurring | Quoted prices in active markets Level 1 | Obligations of foreign governments
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	0	0
Recurring | Quoted prices in active markets Level 1 | Certificates of deposit
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	1,356	1,355
Recurring | Quoted prices in active markets Level 1 | Asset backed securities
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	0	0
Recurring | Quoted prices in active markets Level 1 | Corporate securities
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	0	0
Recurring | Quoted prices in active markets Level 1 | Corporate Loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	0	0
Recurring | Quoted prices in active markets Level 1 | Mortgage Loans Held for Sale
Assets, Fair Value Disclosure [Abstract]
Mortgage loans held for sale	0	0
Recurring | Quoted prices in active markets Level 1 | Equity securities
Assets, Fair Value Disclosure [Abstract]
Equity securities	144,987	123,803
Recurring | Quoted prices in active markets Level 1 | Equity securities | Other equity securities
Assets, Fair Value Disclosure [Abstract]
Equity securities	32,975	28,850
Recurring | Quoted prices in active markets Level 1 | Equity securities | Exchange Traded Funds
Assets, Fair Value Disclosure [Abstract]
Equity securities	64,293	63,875
Recurring | Quoted prices in active markets Level 1 | Equity securities | Invesque Inc.
Assets, Fair Value Disclosure [Abstract]
Equity securities	47,719	31,078
Recurring | Quoted prices in active markets Level 1 | Corporate bonds
Assets, Fair Value Disclosure [Abstract]
Corporate bonds	0	0
Recurring | Quoted prices in active markets Level 1 | CLOs
Assets, Fair Value Disclosure [Abstract]
CLOs	0	0
Recurring | Other significant observable inputs Level 2
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	407,713	374,920
Loans, at fair value	102,773	82,937
Asset derivatives	3,737	232
Total other investments, at fair value	105,033	106,009
Mortgage servicing rights (included in other assets)	0	0
Total assets	615,519	563,866
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	1,095	2,090
Securities sold, not yet purchased	12,573	30,158
Contingent consideration payable	0	0
Total liabilities	13,668	32,248
Recurring | Other significant observable inputs Level 2 | U.S. Treasury securities and obligations of U.S. government authorities and agencies
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	209,735	196,303
Recurring | Other significant observable inputs Level 2 | Obligations of state and political subdivisions
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	47,294	44,350
Recurring | Other significant observable inputs Level 2 | Obligations of foreign governments
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	4,508	3,992
Recurring | Other significant observable inputs Level 2 | Certificates of deposit
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	0	0
Recurring | Other significant observable inputs Level 2 | Asset backed securities
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	37,643	35,334
Recurring | Other significant observable inputs Level 2 | Corporate securities
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	108,533	94,941
Recurring | Other significant observable inputs Level 2 | Corporate Loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	3,147	0
Recurring | Other significant observable inputs Level 2 | Mortgage Loans Held for Sale
Assets, Fair Value Disclosure [Abstract]
Mortgage loans held for sale	99,626	82,937
Recurring | Other significant observable inputs Level 2 | Equity securities
Assets, Fair Value Disclosure [Abstract]
Equity securities	0	0
Recurring | Other significant observable inputs Level 2 | Equity securities | Other equity securities
Assets, Fair Value Disclosure [Abstract]
Equity securities	0	0
Recurring | Other significant observable inputs Level 2 | Equity securities | Exchange Traded Funds
Assets, Fair Value Disclosure [Abstract]
Equity securities	0	0
Recurring | Other significant observable inputs Level 2 | Equity securities | Invesque Inc.
Assets, Fair Value Disclosure [Abstract]
Equity securities	0	0
Recurring | Other significant observable inputs Level 2 | Corporate bonds
Assets, Fair Value Disclosure [Abstract]
Corporate bonds	101,296	105,777
Recurring | Other significant observable inputs Level 2 | CLOs
Assets, Fair Value Disclosure [Abstract]
CLOs	0	0
Recurring | Significant unobservable inputs Level 3
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	878	858
Loans, at fair value	7,685	7,795
Asset derivatives	8,284	9,207
Total other investments, at fair value	8,900	10,009
Mortgage servicing rights (included in other assets)	20,894	14,758
Total assets	38,392	33,455
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	0	0
Securities sold, not yet purchased	0	0
Contingent consideration payable	200	200
Total liabilities	200	200
Recurring | Significant unobservable inputs Level 3 | U.S. Treasury securities and obligations of U.S. government authorities and agencies
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	0	0
Recurring | Significant unobservable inputs Level 3 | Obligations of state and political subdivisions
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	0	0
Recurring | Significant unobservable inputs Level 3 | Obligations of foreign governments
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	0	0
Recurring | Significant unobservable inputs Level 3 | Certificates of deposit
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	0	0
Recurring | Significant unobservable inputs Level 3 | Asset backed securities
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	878	858
Recurring | Significant unobservable inputs Level 3 | Corporate securities
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	0	0
Recurring | Significant unobservable inputs Level 3 | Corporate Loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	7,685	7,795
Recurring | Significant unobservable inputs Level 3 | Mortgage Loans Held for Sale
Assets, Fair Value Disclosure [Abstract]
Mortgage loans held for sale	0	0
Recurring | Significant unobservable inputs Level 3 | Equity securities
Assets, Fair Value Disclosure [Abstract]
Equity securities	35	35
Recurring | Significant unobservable inputs Level 3 | Equity securities | Other equity securities
Assets, Fair Value Disclosure [Abstract]
Equity securities	35	35
Recurring | Significant unobservable inputs Level 3 | Equity securities | Exchange Traded Funds
Assets, Fair Value Disclosure [Abstract]
Equity securities	0	0
Recurring | Significant unobservable inputs Level 3 | Equity securities | Invesque Inc.
Assets, Fair Value Disclosure [Abstract]
Equity securities	0	0
Recurring | Significant unobservable inputs Level 3 | Corporate bonds
Assets, Fair Value Disclosure [Abstract]
Corporate bonds	0	0
Recurring | Significant unobservable inputs Level 3 | CLOs
Assets, Fair Value Disclosure [Abstract]
CLOs	616	802
Fair Value | Other significant observable inputs Level 2
Assets, Fair Value Disclosure [Abstract]
Total assets	83,184	79,778
Fair Value | Significant unobservable inputs Level 3
Liabilities, Fair Value Disclosure [Abstract]
Total liabilities	421,938	392,951
Fair Value | Recurring
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	409,947	377,133
Loans, at fair value	110,458	90,732
Asset derivatives	12,688	11,529
Total other investments, at fair value	114,600	118,108
Mortgage servicing rights (included in other assets)	20,894	14,758
Total assets	800,921	724,569
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	1,095	2,090
Securities sold, not yet purchased	39,625	46,637
Contingent consideration payable	200	200
Total liabilities	40,920	48,927
Fair Value | Recurring | U.S. Treasury securities and obligations of U.S. government authorities and agencies
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	209,735	196,303
Fair Value | Recurring | Obligations of state and political subdivisions
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	47,294	44,350
Fair Value | Recurring | Obligations of foreign governments
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	4,508	3,992
Fair Value | Recurring | Certificates of deposit
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	1,356	1,355
Fair Value | Recurring | Asset backed securities
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	38,521	36,192
Fair Value | Recurring | Corporate securities
Assets, Fair Value Disclosure [Abstract]
Available for sale securities, at fair value	108,533	94,941
Fair Value | Recurring | Corporate Loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	10,832	7,795
Fair Value | Recurring | Mortgage Loans Held for Sale
Assets, Fair Value Disclosure [Abstract]
Mortgage loans held for sale	99,626	82,937
Fair Value | Recurring | Equity securities
Assets, Fair Value Disclosure [Abstract]
Equity securities	145,022	123,838
Fair Value | Recurring | Equity securities | Other equity securities
Assets, Fair Value Disclosure [Abstract]
Equity securities	33,010	28,885
Fair Value | Recurring | Equity securities | Exchange Traded Funds
Assets, Fair Value Disclosure [Abstract]
Equity securities	64,293	63,875
Fair Value | Recurring | Equity securities | Invesque Inc.
Assets, Fair Value Disclosure [Abstract]
Equity securities	47,719	31,078
Fair Value | Recurring | Corporate bonds
Assets, Fair Value Disclosure [Abstract]
Corporate bonds	101,296	105,777
Fair Value | Recurring | CLOs
Assets, Fair Value Disclosure [Abstract]
CLOs	$ 616	$ 802