Fair Value of Financial Instruments - Fair Value Estimates (Details) - USD ($) $ in Thousands	Mar. 31, 2016	Dec. 31, 2015	Mar. 31, 2015
Assets:
Investments held to maturity	$ 9,612	$ 9,572
Investment securities available for sale	269,711	280,273	$ 280,461
Accrued interest receivable	11,862	9,315	8,769
Level 1
Assets:
Cash and cash equivalents and interest bearing deposits with financial institutions	193,514	78,417	178,371
Investments held to maturity	0
Investment securities available for sale	0	$ 0	0
FHLB, FRB, and other stock	0		30,586
Loans held for sale	0	$ 0
Loans held for investment	0	0	0
Accrued interest receivable	11,862	9,315	8,769
Liabilities:
Deposit accounts	2,335,497	1,674,148	1,576,056
FHLB advances	0	0	300,031
Other borrowings	0	0	0
Subordinated debentures	0	0	0
Accrued interest payable	232	206	177
Level 2
Assets:
Cash and cash equivalents and interest bearing deposits with financial institutions	0	0	0
Investments held to maturity	9,590
Investment securities available for sale	269,711	280,273	280,461
FHLB, FRB, and other stock	25,853	22,292	0
Loans held for sale	8,082	9,507
Loans held for investment	0	0	0
Accrued interest receivable	0	0	0
Liabilities:
Deposit accounts	585,822	521,291	479,204
FHLB advances	80,033	148,036	0
Other borrowings	45,764	49,156	45,084
Subordinated debentures	68,093	68,675	69,501
Accrued interest payable	0	0	0
Level 3
Assets:
Cash and cash equivalents and interest bearing deposits with financial institutions	0	0	0
Investments held to maturity	0
Investment securities available for sale	0	0	0
FHLB, FRB, and other stock	0	0	0
Loans held for sale	0	0
Loans held for investment	2,849,797	2,244,936	2,205,161
Accrued interest receivable	0	0	0
Liabilities:
Deposit accounts	0	0	0
FHLB advances	0	0	0
Other borrowings	0	0	0
Subordinated debentures	0	0	0
Accrued interest payable	0	0	0
Carrying Amount
Assets:
Cash and cash equivalents and interest bearing deposits with financial institutions	193,514	78,417	178,371
Investments held to maturity	9,590
Investment securities available for sale	269,711	280,273	280,461
FHLB, FRB, and other stock	25,853	22,292	30,586
Loans held for sale	7,281	8,565
Loans held for investment	2,832,977	2,236,998	2,117,741
Accrued interest receivable	11,862	9,315	8,769
Liabilities:
Deposit accounts	2,906,264	2,195,123	2,043,166
FHLB advances	80,000	148,000	300,000
Other borrowings	44,956	48,125	43,434
Subordinated debentures	70,310	70,310	70,310
Accrued interest payable	232	206	177
Estimated Fair Value
Assets:
Cash and cash equivalents and interest bearing deposits with financial institutions	193,514	78,417	178,371
Investments held to maturity	9,590
Investment securities available for sale	269,711	280,273	280,461
FHLB, FRB, and other stock	25,853	22,292	30,586
Loans held for sale		9,507
Loans held for investment	2,849,797	2,244,936	2,205,161
Accrued interest receivable	11,862	9,315	8,769
Liabilities:
Deposit accounts	2,921,319	2,195,439	2,055,260
FHLB advances	80,033	148,036	300,031
Other borrowings	45,764	49,156	45,084
Subordinated debentures	68,093	68,675	69,501
Accrued interest payable	$ 232	$ 206	$ 177