Loans (Granite Purchased Loans) (Details) (USD $) In Thousands, unless otherwise specified	Mar. 31, 2013	Dec. 31, 2012		Mar. 31, 2012	Dec. 31, 2011
Financing Receivable [Line Items]
Allowance of loan losses	$ 29,641	$ 29,314	[1]	$ 39,795	$ 39,360
Loans held for investment	1,113,765	1,177,035	[1]
PI [Member]
Financing Receivable [Line Items]
Loans held for investment	205,884	228,392
PC [Member]
Financing Receivable [Line Items]
Loans held for investment	907,881	948,643
Granite Corp [Member]
Financing Receivable [Line Items]
Purchased impaired	205,884	228,392		310,389
Purchased contractual	31,580	34,844
Loans held for investment	237,464	263,236
Unpaid principal balance	243,614	271,060
Granite Corp [Member] | Commercial and agricultural [Member]
Financing Receivable [Line Items]
Purchased impaired	12,909	14,666
Purchased contractual	5,701	7,311
Loans held for investment	18,610	21,977
Unpaid principal balance	18,107	21,692
Granite Corp [Member] | Real estate - construction [Member]
Financing Receivable [Line Items]
Purchased impaired	2,328	2,682
Purchased contractual	0	0
Loans held for investment	2,328	2,682
Unpaid principal balance	2,281	2,677
Granite Corp [Member] | Real estate - mortgage, 1-4 family residential [Member]
Financing Receivable [Line Items]
Purchased impaired	36,907	39,215
Purchased contractual	25,879	27,484
Loans held for investment	62,786	66,699
Unpaid principal balance	64,906	69,200
Granite Corp [Member] | Real estate - mortgage, commercial [Member]
Financing Receivable [Line Items]
Purchased impaired	152,437	170,467
Purchased contractual	0	49
Loans held for investment	152,437	170,516
Unpaid principal balance	157,312	176,347
Granite Corp [Member] | Consumer [Member]
Financing Receivable [Line Items]
Purchased impaired	1,303	1,362
Purchased contractual	0	0
Loans held for investment	1,303	1,362
Unpaid principal balance	1,008	1,144
Granite Corp [Member] | PI [Member]
Financing Receivable [Line Items]
Allowance of loan losses	5,400	5,400
Granite Corp [Member] | PC [Member]
Financing Receivable [Line Items]
Allowance of loan losses	$ 800	$ 500

[1] Derived from audited consolidated financial statements