Fair Values - Qualitative Information About Level 3 Fair Value Measurements for Financial Instruments Measured at Fair Value on a Non-recurring Basis (Details) (USD $) In Thousands, unless otherwise specified	12 Months Ended
	Dec. 31, 2013	Dec. 31, 2012
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Impaired loans	$ 33,800	$ 53,900
Nonrecurring basis
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Impaired loans	33,819	53,932
Other real estate owned	18,526	19,926
Commercial real estate | Nonrecurring basis
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Impaired loans	21,100	25,997
Other real estate owned	4,119	3,485
Commercial real estate | Nonrecurring basis | Loans receivable
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Impaired loans	21,100	25,997
Construction real estate | Nonrecurring basis
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Other real estate owned	11,041	12,134
SEPH commercial land and development | Nonrecurring basis
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Impaired loans	4,777
SEPH commercial land and development | Nonrecurring basis | Loans receivable
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Impaired loans	4,777	12,832
Remaining commercial | Nonrecurring basis
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Impaired loans	3,788	8,113
Remaining commercial | Nonrecurring basis | Loans receivable
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Impaired loans	3,788	8,113
Residential real estate | Nonrecurring basis
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Impaired loans	4,154	6,990
Other real estate owned	3,366	4,307
Residential real estate | Nonrecurring basis | Loans receivable
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Impaired loans	4,154	6,990
Commercial real estate | Nonrecurring basis | Other real estate owned
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Other real estate owned	4,119	3,485
Construction real estate | Nonrecurring basis | Other real estate owned
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Other real estate owned	11,041	12,134
Residential Real Estate | Nonrecurring basis | Other real estate owned
Fair Value Inputs, Assets, Quantitative Information [Line Items]
Other real estate owned	$ 3,366	$ 4,307
Sales comparison approach | Commercial real estate | Low | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	0.00%	0.00%
Sales comparison approach | Commercial real estate | High | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	109.00%	116.00%
Sales comparison approach | Commercial real estate | Weighted Average [Member] | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	22.80%	22.30%
Sales comparison approach | SEPH commercial land and development | Low | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	0.00%	0.00%
Sales comparison approach | SEPH commercial land and development | High | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	96.00%	218.00%
Sales comparison approach | SEPH commercial land and development | Weighted Average [Member] | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	13.90%	31.90%
Sales comparison approach | Remaining commercial | Low | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	0.00%	0.00%
Sales comparison approach | Remaining commercial | High | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	40.00%	75.00%
Sales comparison approach | Remaining commercial | Weighted Average [Member] | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	22.40%	26.20%
Sales comparison approach | Residential real estate | Low | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	0.00%	0.00%
Sales comparison approach | Residential real estate | High | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	121.80%	178.00%
Sales comparison approach | Residential real estate | Weighted Average [Member] | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	12.80%	17.90%
Sales comparison approach | Commercial real estate | Low | Nonrecurring basis
Fair Value Inputs [Abstract]
Adj to comparables	0.00%	0.00%
Sales comparison approach | Commercial real estate | High | Nonrecurring basis
Fair Value Inputs [Abstract]
Adj to comparables	140.00%	67.00%
Sales comparison approach | Commercial real estate | Weighted Average [Member] | Nonrecurring basis | Other real estate owned
Fair Value Inputs [Abstract]
Adj to comparables	17.70%	25.80%
Sales comparison approach | Construction real estate | Low | Nonrecurring basis
Fair Value Inputs [Abstract]
Adj to comparables	0.00%	0.00%
Sales comparison approach | Construction real estate | High | Nonrecurring basis
Fair Value Inputs [Abstract]
Adj to comparables	484.00%	273.00%
Sales comparison approach | Construction real estate | Weighted Average [Member] | Nonrecurring basis | Other real estate owned
Fair Value Inputs [Abstract]
Adj to comparables	36.20%	34.00%
Sales comparison approach | Residential Real Estate | Low | Nonrecurring basis
Fair Value Inputs [Abstract]
Adj to comparables	0.00%	1.00%
Sales comparison approach | Residential Real Estate | High | Nonrecurring basis
Fair Value Inputs [Abstract]
Adj to comparables	273.00%	61.00%
Sales comparison approach | Residential Real Estate | Weighted Average [Member] | Nonrecurring basis | Other real estate owned
Fair Value Inputs [Abstract]
Adj to comparables	19.20%	18.00%
Income approach | Commercial real estate | Low | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Capitalization rate	8.00%	7.50%
Income approach | Commercial real estate | High | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Capitalization rate	12.50%	20.90%
Income approach | Commercial real estate | Weighted Average [Member] | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Capitalization rate	9.10%	10.10%
Income approach | Residential real estate | Low | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	7.80%
Income approach | Residential real estate | High | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	10.00%
Income approach | Residential real estate | Weighted Average [Member] | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Adj to comparables	8.40%
Income approach | Commercial real estate | Low | Nonrecurring basis
Fair Value Inputs [Abstract]
Capitalization rate	8.00%	11.00%
Income approach | Commercial real estate | High | Nonrecurring basis
Fair Value Inputs [Abstract]
Capitalization rate	11.50%	11.00%
Income approach | Commercial real estate | Weighted Average [Member] | Nonrecurring basis | Other real estate owned
Fair Value Inputs [Abstract]
Capitalization rate	9.60%	11.00%
Income approach | Construction real estate | Low | Nonrecurring basis
Fair Value Inputs [Abstract]
Capitalization rate		8.50%
Income approach | Construction real estate | High | Nonrecurring basis
Fair Value Inputs [Abstract]
Capitalization rate		8.50%
Income approach | Construction real estate | Weighted Average [Member] | Nonrecurring basis | Other real estate owned
Fair Value Inputs [Abstract]
Capitalization rate		8.50%
Income approach | Residential Real Estate | Low | Nonrecurring basis
Fair Value Inputs [Abstract]
Capitalization rate	5.40%	7.90%
Income approach | Residential Real Estate | High | Nonrecurring basis
Fair Value Inputs [Abstract]
Capitalization rate	7.80%	9.30%
Income approach | Residential Real Estate | Weighted Average [Member] | Nonrecurring basis | Other real estate owned
Fair Value Inputs [Abstract]
Capitalization rate	7.40%	8.70%
Cost approach | Commercial real estate | Low | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Fair Value Inputs, Accumulated Depreciation	11.70%	23.00%
Cost approach | Commercial real estate | High | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Fair Value Inputs, Accumulated Depreciation	65.00%	63.00%
Cost approach | Commercial real estate | Weighted Average [Member] | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Fair Value Inputs, Accumulated Depreciation	37.10%	50.40%
Cost approach | Commercial real estate | Low | Nonrecurring basis
Fair Value Inputs [Abstract]
Fair Value Inputs, Accumulated Depreciation	60.00%	40.90%
Cost approach | Commercial real estate | High | Nonrecurring basis
Fair Value Inputs [Abstract]
Fair Value Inputs, Accumulated Depreciation	95.00%	90.00%
Cost approach | Commercial real estate | Weighted Average [Member] | Nonrecurring basis | Other real estate owned
Fair Value Inputs [Abstract]
Fair Value Inputs, Accumulated Depreciation	80.00%	65.00%
Cost approach | Residential Real Estate | Low | Nonrecurring basis
Fair Value Inputs [Abstract]
Fair Value Inputs, Accumulated Depreciation		6.00%
Cost approach | Residential Real Estate | High | Nonrecurring basis
Fair Value Inputs [Abstract]
Fair Value Inputs, Accumulated Depreciation		6.00%
Cost approach | Residential Real Estate | Weighted Average [Member] | Nonrecurring basis | Other real estate owned
Fair Value Inputs [Abstract]
Fair Value Inputs, Accumulated Depreciation		6.00%
Bulk sale approach | SEPH commercial land and development | Low | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Discount rate	11.00%	11.00%
Bulk sale approach | SEPH commercial land and development | High | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Discount rate	20.00%	55.00%
Bulk sale approach | SEPH commercial land and development | Weighted Average [Member] | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Discount rate	14.90%	23.40%
Bulk sale approach | Remaining commercial | Low | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Discount rate	11.00%	10.00%
Bulk sale approach | Remaining commercial | High | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Discount rate	20.00%	55.00%
Bulk sale approach | Remaining commercial | Weighted Average [Member] | Nonrecurring basis | Loans receivable
Fair Value Inputs [Abstract]
Discount rate	18.00%	18.30%
Bulk sale approach | Commercial real estate | Low | Nonrecurring basis
Fair Value Inputs [Abstract]
Discount rate		13.00%
Bulk sale approach | Commercial real estate | High | Nonrecurring basis
Fair Value Inputs [Abstract]
Discount rate		13.00%
Bulk sale approach | Commercial real estate | Weighted Average [Member] | Nonrecurring basis | Other real estate owned
Fair Value Inputs [Abstract]
Discount rate		13.00%
Bulk sale approach | Construction real estate | Low | Nonrecurring basis
Fair Value Inputs [Abstract]
Discount rate	13.00%	10.00%
Bulk sale approach | Construction real estate | High | Nonrecurring basis
Fair Value Inputs [Abstract]
Discount rate	14.00%	12.00%
Bulk sale approach | Construction real estate | Weighted Average [Member] | Nonrecurring basis | Other real estate owned
Fair Value Inputs [Abstract]
Discount rate	13.60%	10.80%