Loans and Allowance for Loan Losses (Narrative) (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2015 USD ($) loans	Dec. 31, 2014 USD ($) loans	Dec. 31, 2013 USD ($) loans	Dec. 31, 2012 USD ($)
Financing Receivable [Line Items]
Loan increase due to net loan premiums and deferred loan costs	$ 3,700	$ 3,200
Unpaid principal balance of loans serviced for others	304,600	235,000
Investment securities pledged to collateralize FHLB advances and letters of credit	113,000	124,600
Gross loans pledged to collateralize FHLB advances and letters of credit	102,900	127,200
Credit availability for borrowing under FHLB advances and letters of credit	80,800	130,800
Securities pledged to collateralize potential borrowings from Federal Reserve Discount Window	44,700
Loans pledged to collateralize potential borrowings from Federal Reserve Discount Window	4,100
Borrowing capacity under Federal Reserve Discount Window	46,600
Additional borrowings from the FHLB	294,100
Interest lost on nonaccrual loans	1,400	3,700	$ 4,700
Impaired loans on nonaccruing interest status	18,000	25,300
Restructured loans	17,900	19,400
Allowance of loan losses	15,195	20,345	26,785	$ 29,314
Charge-offs	(5,604)	(7,703)	(13,344)
Recoveries	3,435	6,634	10,292
Net charge-offs	$ (2,169)	$ (1,069)	$ (3,052)
Percentage of loans held for investment	0.98%	1.50%	2.21%
Number of modified loans | loans	12	25	19
Number of modified loans with extended terms | loans	5	8
Number of modified loans with extended terms and modified interest rates | loans	7	17
Reserve for unfunded commitments	$ 800	$ 800
Granite Corp. [Member]
Financing Receivable [Line Items]
Net carrying amount, end of period	84,810	119,605
PC [Member]
Financing Receivable [Line Items]
Restructured loans	13,100	14,100
Substandard or Doubtful [Member]
Financing Receivable [Line Items]
Net carrying amount, end of period	27,000	40,500
PI loans [Member] | Granite Corp. [Member]
Financing Receivable [Line Items]
Allowance of loan losses	2,800	3,200
PC and Originated Loans [Member] | Granite Corp. [Member]
Financing Receivable [Line Items]
Allowance of loan losses	$ 300	300
Real estate - mortgage, 1-4 family residential [Member] | PC [Member] | Granite Corp. [Member]
Financing Receivable [Line Items]
Loans purchases		19,700	$ 157,600
Loans purchases premium		$ 300	$ 1,300
Builder Guidance Lines of Credit [Member]
Financing Receivable [Line Items]
Expected usage percentage of remaining unfunded commitments	50.00%
Line of Credit [Member]
Financing Receivable [Line Items]
Expected usage percentage of remaining unfunded commitments	100.00%
Letter of Credit [Member]
Financing Receivable [Line Items]
Expected usage percentage of remaining unfunded commitments	10.00%