Goodwill and Intangible Assets (Details)	12 Months Ended
	Dec. 31, 2017 USD ($) $ / loans Rate	Dec. 31, 2016 USD ($) $ / loans Rate	Dec. 31, 2015 USD ($)	Dec. 31, 2014 USD ($)
Servicing Asset at Amortized Cost [Line Items]
Servicing Asset at Amortized Cost	$ 21,945,000	$ 20,415,000	$ 19,351,000	$ 16,688,000
Servicing Asset at Amortized Cost, Additions	7,174,000	5,931,000	5,910,000
Amortization	(5,648,000)	(4,958,000)	(4,002,000)
Valuation Allowance for Impairment of Recognized Servicing Assets, Provisions (Recoveries)	(4,000)	(91,000)	(755,000)
Valuation Allowance for Impairment of Recognized Servicing Assets, Balance	0	4,000	95,000	$ 850,000
Residential Mortgage Loans Serviced	$ 2,810,000,000	$ 2,490,000,000	2,150,000,000
Servicing Assets and Servicing Liabilities at Fair Value, Assumptions Used to Estimate Fair Value, Prepayment Speed | Rate	10.93%	10.42%
Contractually Specified Servicing Fee, Late Fee, and Ancillary Fee Earned in Exchange for Servicing Financial Asset	$ 7,100,000	$ 5,800,000	5,400,000
Servicing Assets and Servicing Liabilities at Fair Value, Assumptions Used to Estimate Fair Value, Cost to Service Loans | $ / loans	64.03	62.75
Goodwill [Roll Forward]
Beginning goodwill	$ 150,601,000	$ 139,773,000
Ending goodwill	150,601,000	150,601,000	139,773,000
Finite-lived Intangible Assets [Roll Forward]
Beginning balance	57,625,000	71,635,000
Amortization	(17,194,000)	(16,850,000)
Ending balance	51,151,000	57,625,000	$ 71,635,000
Finite-Lived Intangible Assets [Abstract]
Gross balance	128,761,000	118,041,000
Accumulated amortization	(77,610,000)	(60,416,000)
Carrying value	51,151,000	57,625,000
Finite-Lived Intangible Assets, Net, Amortization Expense, Fiscal Year Maturity [Abstract]
2018	15,394,000
2019	12,275,000
2020	9,431,000
2021	6,799,000
2022	4,288,000
North Milwaukee State Bank
Finite-lived Intangible Assets [Roll Forward]
Intangible assets acquired	0	240,000
First CornerStone Bank
Finite-lived Intangible Assets [Roll Forward]
Intangible assets acquired	0	390,000
Cordia Bancorp Inc.
Goodwill [Roll Forward]
Goodwill acquired	0	10,828,000
Finite-lived Intangible Assets [Roll Forward]
Intangible assets acquired	0	2,210,000
Harvest Community Bank
Finite-lived Intangible Assets [Roll Forward]
Intangible assets acquired	850,000	0
Guaranty Bank
Finite-lived Intangible Assets [Roll Forward]
Intangible assets acquired	$ 9,870,000	$ 0
Conventional fixed loans
Servicing Asset at Amortized Cost [Line Items]
Servicing Assets and Servicing Liabilities at Fair Value, Assumptions Used to Estimate Fair Value, Discount Rate | Rate	9.41%	9.45%
All loans and leases excluding conventional fixed loans [Member]
Servicing Asset at Amortized Cost [Line Items]
Servicing Assets and Servicing Liabilities at Fair Value, Assumptions Used to Estimate Fair Value, Discount Rate | Rate	10.41%	10.45%
Core deposits | Minimum
Goodwill [Roll Forward]
Useful life	1 year
Core deposits | Maximum
Goodwill [Roll Forward]
Useful life	8 years