Loans (Loan Portfolio by Class According to Credit Quality Indicators) (Details) - USD ($) $ in Thousands	Dec. 31, 2017	Dec. 31, 2016
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 1,147,864	$ 958,641
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	43,999	42,492
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	21,871	26,344
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	606,994	509,171
Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	82,825	58,447
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	469,696	422,418
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	54,473	28,306
SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	672,864	578,007
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	365,145	289,093
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	109,855	91,541
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	55,817	47,411
Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	54,038	44,130
Residential And Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	475,000	380,634
Pass | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	42,415	38,990
Pass | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	20,751	24,635
Pass | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	601,263	492,844
Pass | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	82,201	57,000
Pass | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	464,589	408,288
Pass | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	54,473	27,556
Pass | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	664,429	556,469
Special mention | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	373	2,023
Special mention | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,024	1,073
Special mention | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,646	15,901
Special mention | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	599	1,422
Special mention | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,047	13,729
Special mention | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	750
Special mention | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	5,043	18,997
Substandard | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,211	1,479
Substandard | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	96	636
Substandard | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,085	426
Substandard | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	25	25
Substandard | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,060	401
Substandard | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Substandard | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,392	2,541
Performing | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	363,476	286,421
Performing | Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	109,230	890,833
Performing | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	55,192	46,929
Performing | Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	54,038	42,154
Performing | Residential And Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	472,706	375,504
Nonperforming | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,669	2,672
Nonperforming | Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	625	2,458
Nonperforming | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	625	482
Nonperforming | Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	1,976
Nonperforming | Residential And Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 2,294	$ 5,130