Loans and Credit Quality - Loans by Loan Grade (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	$ 3,986,180	$ 3,849,451
Pass [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	3,818,954	3,666,442
Watch [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	115,711	129,445
Special Mention [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	27,731	29,827
Substandard [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	23,784	23,737
Consumer Loans [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	1,481,084	1,443,696
Consumer Loans [Member] | Pass [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	1,446,295	1,408,654
Consumer Loans [Member] | Watch [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	3,264	4,945
Consumer Loans [Member] | Special Mention [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	16,653	15,686
Consumer Loans [Member] | Substandard [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	14,872	14,411
Consumer Loans [Member] | Single Family [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	1,100,215	1,083,822
Consumer Loans [Member] | Single Family [Member] | Pass [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	1,067,886	1,051,463
Consumer Loans [Member] | Single Family [Member] | Watch [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	3,109	4,348
Consumer Loans [Member] | Single Family [Member] | Special Mention [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	16,203	15,172
Consumer Loans [Member] | Single Family [Member] | Substandard [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	13,017	12,839
Consumer Loans [Member] | Home Equity and Other [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	380,869	359,874
Consumer Loans [Member] | Home Equity and Other [Member] | Pass [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	378,409	357,191
Consumer Loans [Member] | Home Equity and Other [Member] | Watch [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	155	597
Consumer Loans [Member] | Home Equity and Other [Member] | Special Mention [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	450	514
Consumer Loans [Member] | Home Equity and Other [Member] | Substandard [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	1,855	1,572
Commercial Loans [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	2,505,096	2,405,755
Commercial Loans [Member] | Pass [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	2,372,659	2,257,788
Commercial Loans [Member] | Watch [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	112,447	124,500
Commercial Loans [Member] | Special Mention [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	11,078	14,141
Commercial Loans [Member] | Substandard [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	8,912	9,326
Commercial Loans [Member] | Commercial Real Estate [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	922,852	871,563
Commercial Loans [Member] | Commercial Real Estate [Member] | Pass [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	872,245	809,996
Commercial Loans [Member] | Commercial Real Estate [Member] | Watch [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	39,979	52,519
Commercial Loans [Member] | Commercial Real Estate [Member] | Special Mention [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	5,251	7,165
Commercial Loans [Member] | Commercial Real Estate [Member] | Substandard [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	5,377	1,883
Commercial Loans [Member] | Multifamily [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	748,333	674,219
Commercial Loans [Member] | Multifamily [Member] | Pass [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	734,679	660,234
Commercial Loans [Member] | Multifamily [Member] | Watch [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	11,430	13,140
Commercial Loans [Member] | Multifamily [Member] | Special Mention [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	1,896	508
Commercial Loans [Member] | Multifamily [Member] | Substandard [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	328	337
Commercial Loans [Member] | Construction/Land Development [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	611,150	636,320
Commercial Loans [Member] | Construction/Land Development [Member] | Pass [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	591,963	615,675
Commercial Loans [Member] | Construction/Land Development [Member] | Watch [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	17,802	16,074
Commercial Loans [Member] | Construction/Land Development [Member] | Special Mention [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	1,013	3,083
Commercial Loans [Member] | Construction/Land Development [Member] | Substandard [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	372	1,488
Commercial Loans [Member] | Commercial Business [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	222,761	223,653
Commercial Loans [Member] | Commercial Business [Member] | Pass [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	173,772	171,883
Commercial Loans [Member] | Commercial Business [Member] | Watch [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	43,236	42,767
Commercial Loans [Member] | Commercial Business [Member] | Special Mention [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	2,918	3,385
Commercial Loans [Member] | Commercial Business [Member] | Substandard [Member]
Designated Loan Grades by Loan Portfolio Segment and Loan Class [Abstract]
Total loans	$ 2,835	$ 5,618