Loans (Loan Portfolio by Class According to Credit Quality Indicators) (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 1,412,029	$ 1,293,395
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	35,767	39,333
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	26,726	29,155
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	765,032	694,102
Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	112,014	104,587
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	578,643	510,370
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	47,649	49,990
Total SBA and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	800,799	733,435
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	467,706	436,056
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	143,524	123,904
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	69,589	59,887
Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	73,935	64,017
Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	611,230	559,960
Pass | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	34,202	37,198
Pass | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	24,878	28,105
Pass | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	756,073	685,923
Pass | Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	107,220	103,806
Pass | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	576,326	504,022
Pass | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	47,649	49,990
Pass | Total SBA and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	790,275	723,121
Special mention | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,115	601
Special mention | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,808	0
Special mention | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	5,927	3,201
Special mention | Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,361	322
Special mention | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	758	2,879
Special mention | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Special mention | Total SBA and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	7,042	3,802
Substandard | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	450	1,534
Substandard | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	40	1,050
Substandard | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,032	4,978
Substandard | Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,433	459
Substandard | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,559	3,469
Substandard | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Substandard | Total SBA and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,482	6,512
Performing | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	463,770	431,845
Performing | Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	143,504	123,878
Performing | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	69,589	59,861
Performing | Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	73,915	64,017
Performing | Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	607,274	555,723
Nonperforming | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,936	4,211
Nonperforming | Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	20	26
Nonperforming | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	26
Nonperforming | Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	20	0
Nonperforming | Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 3,956	$ 4,237