Fair Value of Financial Instruments - Schedule of Fair Value Hierarchies for Financial Assets and Liabilities Measured on a Recurring Basis (Details) - USD ($) $ in Thousands	9 Months Ended
	Sep. 30, 2018	Dec. 31, 2017
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Average Estimated Restriction, Period	5 months
Foreclosed residential real estate property	$ 11,746	$ 16,056
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	255,784	182,448
Loans, at fair value	229,033	258,173
Equity securities, at fair value	135,223	25,536
Asset derivatives	4,820	5,013
Total other investments, at fair value	71,909	59,142
Liabilities, Fair Value Disclosure [Abstract]
Liability derivatives	77	117
Significant unobservable inputs Level 3
Assets, Fair Value Disclosure [Abstract]
Total assets	33,086	42,474
Significant unobservable inputs Level 3 | Non-performing loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	28,693	37,666
Significant unobservable inputs Level 3 | Interest Rate Lock Commitments
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	4,393	4,808
Recurring | Quoted prices in active markets Level 1
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	2,296	1,437
Loans, at fair value	0	0
Asset derivatives	0	0
Total other investments, at fair value	0	0
Total assets	13,353	26,973
Liabilities, Fair Value Disclosure [Abstract]
Total derivative liabilities (included in other liabilities and accrued expenses)	0	0
Total liabilities	0	0
Recurring | Quoted prices in active markets Level 1 | Interest Rate Lock Commitments
Liabilities, Fair Value Disclosure [Abstract]
Liability derivatives	0	0
Recurring | Quoted prices in active markets Level 1 | Contingent Consideration Payable
Liabilities, Fair Value Disclosure [Abstract]
Contingent consideration payable	0	0
Recurring | Quoted prices in active markets Level 1 | U.S. Treasury securities and obligations of U.S. government authorities and agencies
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	0	0
Recurring | Quoted prices in active markets Level 1 | Obligations of states and political subdivisions
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	0	0
Recurring | Quoted prices in active markets Level 1 | Obligations of foreign governments
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	0	0
Recurring | Quoted prices in active markets Level 1 | Certificates of deposit
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	2,296	896
Recurring | Quoted prices in active markets Level 1 | Asset-backed Securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	0	0
Recurring | Quoted prices in active markets Level 1 | Corporate securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	0	0
Recurring | Quoted prices in active markets Level 1 | Corporate Loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	0	0
Recurring | Quoted prices in active markets Level 1 | Mortgage Loans Held for Sale
Assets, Fair Value Disclosure [Abstract]
Mortgage loans held for sale	0	0
Recurring | Quoted prices in active markets Level 1 | Non-performing loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	0	0
Recurring | Quoted prices in active markets Level 1 | Equity securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities		541
Equity securities, at fair value	11,057	25,536
Recurring | Quoted prices in active markets Level 1 | Forward Delivery Contracts
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	0	0
Recurring | Quoted prices in active markets Level 1 | Interest Rate Lock Commitments
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	0	0
Recurring | Quoted prices in active markets Level 1 | CLOs
Assets, Fair Value Disclosure [Abstract]
Trading assets	0	0
Recurring | Quoted prices in active markets Level 1 | Debentures
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	0	0
Recurring | Other significant observable inputs Level 2
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	250,535	180,964
Loans, at fair value	90,587	103,771
Asset derivatives	427	205
Total other investments, at fair value	5,582	4,368
Total assets	470,553	289,103
Liabilities, Fair Value Disclosure [Abstract]
Total derivative liabilities (included in other liabilities and accrued expenses)	77	117
Total liabilities	77	117
Recurring | Other significant observable inputs Level 2 | Interest Rate Lock Commitments
Liabilities, Fair Value Disclosure [Abstract]
Liability derivatives	77	117
Recurring | Other significant observable inputs Level 2 | Contingent Consideration Payable
Liabilities, Fair Value Disclosure [Abstract]
Contingent consideration payable	0	0
Recurring | Other significant observable inputs Level 2 | U.S. Treasury securities and obligations of U.S. government authorities and agencies
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	54,247	47,945
Recurring | Other significant observable inputs Level 2 | Obligations of states and political subdivisions
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	61,161	46,981
Recurring | Other significant observable inputs Level 2 | Obligations of foreign governments
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	6,529	570
Recurring | Other significant observable inputs Level 2 | Certificates of deposit
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	0	0
Recurring | Other significant observable inputs Level 2 | Asset-backed Securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	42,616	23,493
Recurring | Other significant observable inputs Level 2 | Corporate securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	85,982	61,975
Recurring | Other significant observable inputs Level 2 | Corporate Loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	39,839	40,925
Recurring | Other significant observable inputs Level 2 | Mortgage Loans Held for Sale
Assets, Fair Value Disclosure [Abstract]
Mortgage loans held for sale	50,748	62,846
Recurring | Other significant observable inputs Level 2 | Non-performing loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	0	0
Recurring | Other significant observable inputs Level 2 | Equity securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities		0
Equity securities, at fair value	123,849	0
Recurring | Other significant observable inputs Level 2 | Forward Delivery Contracts
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	16	30
Recurring | Other significant observable inputs Level 2 | Interest Rate Lock Commitments
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	0	0
Recurring | Other significant observable inputs Level 2 | CLOs
Assets, Fair Value Disclosure [Abstract]
Trading assets	0	0
Recurring | Other significant observable inputs Level 2 | Debentures
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	5,155	4,163
Recurring | Significant unobservable inputs Level 3
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	2,953	47
Loans, at fair value	138,446	154,402
Asset derivatives	4,393	4,808
Total other investments, at fair value	6,303	8,217
Total assets	148,019	162,666
Liabilities, Fair Value Disclosure [Abstract]
Total derivative liabilities (included in other liabilities and accrued expenses)	0	0
Total liabilities	0	0
Recurring | Significant unobservable inputs Level 3 | Interest Rate Lock Commitments
Liabilities, Fair Value Disclosure [Abstract]
Liability derivatives	0	0
Recurring | Significant unobservable inputs Level 3 | Contingent Consideration Payable
Liabilities, Fair Value Disclosure [Abstract]
Contingent consideration payable	0	0
Recurring | Significant unobservable inputs Level 3 | U.S. Treasury securities and obligations of U.S. government authorities and agencies
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	0	0
Recurring | Significant unobservable inputs Level 3 | Obligations of states and political subdivisions
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	0	0
Recurring | Significant unobservable inputs Level 3 | Obligations of foreign governments
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	0	0
Recurring | Significant unobservable inputs Level 3 | Certificates of deposit
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	0	0
Recurring | Significant unobservable inputs Level 3 | Asset-backed Securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	2,953	0
Recurring | Significant unobservable inputs Level 3 | Corporate securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	0	0
Recurring | Significant unobservable inputs Level 3 | Corporate Loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	109,753	116,736
Recurring | Significant unobservable inputs Level 3 | Mortgage Loans Held for Sale
Assets, Fair Value Disclosure [Abstract]
Mortgage loans held for sale	0	0
Recurring | Significant unobservable inputs Level 3 | Non-performing loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	28,693	37,666
Recurring | Significant unobservable inputs Level 3 | Equity securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities		47
Equity securities, at fair value	317	0
Recurring | Significant unobservable inputs Level 3 | Forward Delivery Contracts
Assets, Fair Value Disclosure [Abstract]
Asset derivatives		0
Recurring | Significant unobservable inputs Level 3 | Interest Rate Lock Commitments
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	4,393	4,808
Recurring | Significant unobservable inputs Level 3 | CLOs
Assets, Fair Value Disclosure [Abstract]
Trading assets	1,910	3,409
Recurring | Significant unobservable inputs Level 3 | Debentures
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	0	0
Fair Value | Other significant observable inputs Level 2
Assets, Fair Value Disclosure [Abstract]
Total assets	12,375	12,225
Fair Value | Significant unobservable inputs Level 3
Liabilities, Fair Value Disclosure [Abstract]
Total liabilities	376,960	356,537
Fair Value | Recurring
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	255,784	182,448
Loans, at fair value	229,033	258,173
Asset derivatives	4,820	5,013
Total other investments, at fair value	11,885	12,585
Total assets	631,925	478,742
Liabilities, Fair Value Disclosure [Abstract]
Total derivative liabilities (included in other liabilities and accrued expenses)	77	117
Total liabilities	77	117
Fair Value | Recurring | Interest Rate Lock Commitments
Liabilities, Fair Value Disclosure [Abstract]
Liability derivatives	77	117
Fair Value | Recurring | Contingent Consideration Payable
Liabilities, Fair Value Disclosure [Abstract]
Contingent consideration payable	0
Fair Value | Recurring | U.S. Treasury securities and obligations of U.S. government authorities and agencies
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	54,247	47,945
Fair Value | Recurring | Obligations of states and political subdivisions
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	61,161	46,981
Fair Value | Recurring | Obligations of foreign governments
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	6,529	570
Fair Value | Recurring | Certificates of deposit
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	2,296	896
Fair Value | Recurring | Asset-backed Securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	45,569	23,493
Fair Value | Recurring | Corporate securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities	85,982	61,975
Fair Value | Recurring | Corporate Loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	149,592	157,661
Fair Value | Recurring | Mortgage Loans Held for Sale
Assets, Fair Value Disclosure [Abstract]
Mortgage loans held for sale	50,748	62,846
Fair Value | Recurring | Non-performing loans
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	28,693	37,666
Fair Value | Recurring | Equity securities
Assets, Fair Value Disclosure [Abstract]
Investments in available for sale securities		588
Equity securities, at fair value	135,223	25,536
Fair Value | Recurring | Forward Delivery Contracts
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	16	30
Fair Value | Recurring | Interest Rate Lock Commitments
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	4,393	4,808
Fair Value | Recurring | CLOs
Assets, Fair Value Disclosure [Abstract]
Trading assets	1,910	3,409
Fair Value | Recurring | Debentures
Assets, Fair Value Disclosure [Abstract]
Loans, at fair value	5,155	4,163
TBA mortgage backed securities | Recurring | Quoted prices in active markets Level 1 | Interest Rate Lock Commitments
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	0	0
TBA mortgage backed securities | Recurring | Other significant observable inputs Level 2 | Interest Rate Lock Commitments
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	411	175
TBA mortgage backed securities | Recurring | Significant unobservable inputs Level 3 | Interest Rate Lock Commitments
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	0	0
TBA mortgage backed securities | Fair Value | Recurring | Interest Rate Lock Commitments
Assets, Fair Value Disclosure [Abstract]
Asset derivatives	$ 411	$ 175