Loans and Allowance for Credit Losses on Loans - Loans Balance (Details) - USD ($)	Dec. 31, 2023	Dec. 31, 2022	Dec. 31, 2021	Dec. 31, 2020
Loans held-for-investment:
Total loan balance	$ 3,350,873,000	$ 3,299,067,000
Deferred loan fees, net	(495,000)	(517,000)
Loans, net of deferred fees	3,350,378,000	3,298,550,000	$ 3,087,326,000
Allowance for credit losses on loans	(47,958,000)	(47,512,000)	(43,290,000)	$ (44,400,000)
Loans, net	3,302,420,000	3,251,038,000
Commercial. | Commercial
Loans held-for-investment:
Total loan balance	463,778,000	533,915,000
Allowance for credit losses on loans	(5,853,000)	(6,617,000)	(8,414,000)	(11,587,000)
Real estate | CRE - Owner Occupied
Loans held-for-investment:
Total loan balance	583,253,000	614,663,000
Allowance for credit losses on loans	(5,121,000)	(5,751,000)	(7,954,000)	(8,560,000)
Real estate | CRE - Non-Owner Occupied
Loans held-for-investment:
Total loan balance	1,256,590,000	1,066,368,000
Allowance for credit losses on loans	(25,323,000)	(22,135,000)	(17,125,000)	(16,416,000)
Real estate | Land & Construction
Loans held-for-investment:
Total loan balance	140,513,000	163,577,000
Allowance for credit losses on loans	(2,352,000)	(2,941,000)	(1,831,000)	(2,509,000)
Real estate | Home Equity
Loans held-for-investment:
Total loan balance	119,125,000	120,724,000
Allowance for credit losses on loans	(644,000)	(666,000)	(864,000)	(1,297,000)
Real estate | Multi-Family
Loans held-for-investment:
Total loan balance	269,734,000	244,882,000
Allowance for credit losses on loans	(5,053,000)	(3,366,000)	(2,796,000)	(2,804,000)
Real estate | Residential Mortgage
Loans held-for-investment:
Total loan balance	496,961,000	537,905,000
Allowance for credit losses on loans	(3,425,000)	(5,907,000)	(4,132,000)	(943,000)
Consumer | Consumer and Other
Loans held-for-investment:
Total loan balance	20,919,000	17,033,000
Allowance for credit losses on loans	$ (187,000)	$ (129,000)	$ (174,000)	$ (284,000)