Fair Value Measurements - Estimated Fair Value (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Financial assets:
Held to maturity securities	$ 345,088	$ 331,468
Non-covered loans, net	5,789,313	5,162,202
Covered loans, net	255,714	378,762
Broker-dealer and clearing organization receivables	1,497,741	1,362,499
FDIC indemnification asset	71,313	91,648	$ 130,437	$ 188,291
Financial liabilities:
Broker-dealer and clearing organization payables	1,347,128	1,338,305
Carrying Amount
Financial assets:
Cash and cash equivalents	690,764	669,445
Securities purchased under agreements to resell	89,430	105,660
Assets segregated for regulatory purposes	180,993	158,613
Held to maturity securities	351,831	332,022
Loans held for sale	46,965	72,843
Non-covered loans, net	5,789,313	5,162,202
Covered loans, net	255,714	378,762
Broker-dealer and clearing organization receivables	1,497,741	1,362,499
FDIC indemnification asset	71,313	91,648
Other assets	62,904	68,786
Financial liabilities:
Deposits	7,063,811	6,952,683
Broker-dealer and clearing organization payables	1,347,128	1,338,305
Short-term borrowings	1,417,289	947,373
Debt	384,924	305,728
Other liabilities	3,708	3,699
Estimate of Fair Value
Financial assets:
Cash and cash equivalents	690,764	669,445
Securities purchased under agreements to resell	89,430	105,660
Assets segregated for regulatory purposes	180,993	158,613
Held to maturity securities	345,088	331,468
Loans held for sale	46,965	72,843
Non-covered loans, net	5,962,052	5,190,951
Covered loans, net	367,444	527,201
Broker-dealer and clearing organization receivables	1,497,741	1,362,499
FDIC indemnification asset	60,173	91,648
Other assets	62,904	68,786
Financial liabilities:
Deposits	7,058,837	6,955,919
Broker-dealer and clearing organization payables	1,347,128	1,338,305
Short-term borrowings	1,417,289	947,373
Debt	378,822	299,257
Other liabilities	3,708	3,699
Estimate of Fair Value | Level 1
Financial assets:
Cash and cash equivalents	690,764	669,445
Assets segregated for regulatory purposes	180,993	158,613
Estimate of Fair Value | Level 2
Financial assets:
Securities purchased under agreements to resell	89,430	105,660
Held to maturity securities	345,088	331,468
Loans held for sale	46,965	72,843
Non-covered loans, net	502,077	590,545
Broker-dealer and clearing organization receivables	1,497,741	1,362,499
Other assets	58,697	53,214
Financial liabilities:
Deposits	7,058,837	6,955,919
Broker-dealer and clearing organization payables	1,347,128	1,338,305
Short-term borrowings	1,417,289	947,373
Debt	378,822	299,257
Other liabilities	3,708	3,699
Estimate of Fair Value | Level 3
Financial assets:
Non-covered loans, net	5,459,975	4,600,406
Covered loans, net	367,444	527,201
FDIC indemnification asset	60,173	91,648
Other assets	$ 4,207	$ 15,572