Loans and Allowance for Credit Losses on Loans - Loans Balance (Details)	3 Months Ended
	Jun. 30, 2021 USD ($) loan	Mar. 31, 2021 USD ($)	Dec. 31, 2020 USD ($)	Jun. 30, 2020 USD ($)	Mar. 31, 2020 USD ($)	Jan. 01, 2020 USD ($)	Dec. 31, 2019 USD ($)
Loans held-for-investment:
Total loan balance	$ 2,832,866,000		$ 2,625,987,000
Deferred loan fees, net	(8,070,000)		(6,726,000)
Loans, net of deferred fees	2,824,796,000		2,619,261,000	$ 2,686,389,000
Allowance for credit losses on loans	(43,956,000)	$ (44,296,000)	(44,400,000)	(45,444,000)	$ (44,703,000)	$ (31,855,000)	$ (23,285,000)
Loans, net	2,780,840,000		2,574,861,000
Commercial | SBA Paycheck Protection Program ("PPP")
Loans held-for-investment:
Allowance for credit losses on loans	0		0
Loans, net	286,461,000		290,679,000
Commercial
Loans held-for-investment:
Total loan balance	844,147,000		846,386,000
Allowance for credit losses on loans	(10,857,000)	(11,600,000)	(11,587,000)	(13,179,000)	(12,801,000)
Commercial | Commercial
Loans held-for-investment:
Total loan balance	844,147,000		846,386,000
Commercial Loans excluding SBA Paycheck Protection Program [Member] | Commercial
Loans held-for-investment:
Total loan balance	844,147,000		846,386,000
CRE
Loans held-for-investment:
Allowance for credit losses on loans				(13,179,000)		(6,790,000)	(10,453,000)
CRE - owner occupied
Loans held-for-investment:
Total loan balance	583,091,000		560,362,000
Allowance for credit losses on loans	(8,206,000)	(8,368,000)	(8,560,000)	(8,547,000)	(7,737,000)	(6,994,000)	(3,825,000)
CRE - owner occupied | Real estate
Loans held-for-investment:
Total loan balance	583,091,000		560,362,000
CRE - non-owner occupied
Loans held-for-investment:
Total loan balance	742,135,000		693,103,000
Allowance for credit losses on loans	(16,485,000)	(16,431,000)	(16,416,000)	(15,449,000)	(15,645,000)	(11,672,000)	(3,760,000)
CRE - non-owner occupied | Real estate
Loans held-for-investment:
Total loan balance	742,135,000		693,103,000
Land and construction
Loans held-for-investment:
Total loan balance	129,426,000		144,594,000
Allowance for credit losses on loans	(2,136,000)	(2,754,000)	(2,509,000)	(2,552,000)	(2,603,000)	(1,458,000)	(2,621,000)
Land and construction | Real estate
Loans held-for-investment:
Total loan balance	129,426,000		144,594,000
Home equity
Loans held-for-investment:
Total loan balance	107,873,000		111,885,000
Allowance for credit losses on loans	(1,069,000)	(1,171,000)	(1,297,000)	(1,851,000)	(1,746,000)	(1,321,000)	(2,244,000)
Home equity | Real estate
Loans held-for-investment:
Total loan balance	107,873,000		111,885,000
Multi-family
Loans held-for-investment:
Total loan balance	198,771,000		166,425,000
Allowance for credit losses on loans	(2,950,000)	(2,751,000)	(2,804,000)	(1,828,000)	(1,622,000)	(1,253,000)	(57,000)
Multi-family | Real estate
Loans held-for-investment:
Total loan balance	198,771,000		166,425,000
Residential mortgages
Loans held-for-investment:
Total loan balance	205,904,000		85,116,000
Allowance for credit losses on loans	$ (1,968,000)	(918,000)	(943,000)	(825,000)	(708,000)	(678,000)	(243,000)
Number of loan portfolios purchased | loan	2
Value of loan portfolios purchased	$ 140,030,000
Average principal balance	$ 585,000
Weighted average yield	3.39%
Residential mortgages | Real estate
Loans held-for-investment:
Total loan balance	$ 205,904,000		85,116,000
Consumer and other
Loans held-for-investment:
Total loan balance	21,519,000		18,116,000
Allowance for credit losses on loans	(285,000)	$ (303,000)	(284,000)	$ (1,213,000)	$ (1,841,000)	$ (1,689,000)	(82,000)
Consumer and other | Consumer
Loans held-for-investment:
Total loan balance	$ 21,519,000		$ 18,116,000
ASU 2016-03 - Topic 326
Loans held-for-investment:
Allowance for credit losses on loans							(8,570,000)
ASU 2016-03 - Topic 326 | CRE
Loans held-for-investment:
Allowance for credit losses on loans							3,663,000
ASU 2016-03 - Topic 326 | CRE - owner occupied
Loans held-for-investment:
Allowance for credit losses on loans							(3,169,000)
ASU 2016-03 - Topic 326 | CRE - non-owner occupied
Loans held-for-investment:
Allowance for credit losses on loans							(7,912,000)
ASU 2016-03 - Topic 326 | Land and construction
Loans held-for-investment:
Allowance for credit losses on loans							1,163,000
ASU 2016-03 - Topic 326 | Home equity
Loans held-for-investment:
Allowance for credit losses on loans							923,000
ASU 2016-03 - Topic 326 | Multi-family
Loans held-for-investment:
Allowance for credit losses on loans							(1,196,000)
ASU 2016-03 - Topic 326 | Residential mortgages
Loans held-for-investment:
Allowance for credit losses on loans							(435,000)
ASU 2016-03 - Topic 326 | Consumer and other
Loans held-for-investment:
Allowance for credit losses on loans							$ (1,607,000)