Description of Business and Summary of Significant Accounting Policies - Impact of adoption of the CECL model Under ASC 326 (Details) - USD ($)	Dec. 31, 2020	Jan. 01, 2020	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on debt securities, held-to-maturity	$ 0	$ 0
Allowance for credit losses on debt securities, available-for-sale	0	0
Allowance for credit losses on loans	268,018,000		$ 35,698,000	$ 36,072,000	$ 28,936,000
Deferred tax (liabilities) assets		(1,371,000)
Allowance for credit losses on off-balance sheet credit exposures	31,100,000	3,279,000	3,300,000
Retained earnings	$ 330,555,000	396,051,000	396,051,000
Investor loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		9,027,000
Business loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		5,492,000
Commercial loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		20,118,000
Retail loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		1,061,000
Impact of CECL Adoption
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on debt securities, held-to-maturity		0
Allowance for credit losses on debt securities, available-for-sale		0
Allowance for credit losses on loans		64,000,000.0	$ 55,686,000
Deferred tax (liabilities) assets		18,346,000
Allowance for credit losses on off-balance sheet credit exposures		8,285,000
Retained earnings		(45,625,000)
Impact of CECL Adoption | Investor loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		16,072,000
Impact of CECL Adoption | Business loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		27,572,000
Impact of CECL Adoption | Commercial loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		9,519,000
Impact of CECL Adoption | Retail loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		2,523,000
As Reported Under CECL
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on debt securities, held-to-maturity		0
Allowance for credit losses on debt securities, available-for-sale		0
Deferred tax (liabilities) assets		16,975,000
Allowance for credit losses on off-balance sheet credit exposures		11,564,000
Retained earnings		350,426,000
As Reported Under CECL | Investor loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		25,099,000
As Reported Under CECL | Business loans secured by real estate
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		33,064,000
As Reported Under CECL | Commercial loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		29,637,000
As Reported Under CECL | Retail loans
New Accounting Pronouncements or Change in Accounting Principle [Line Items]
Allowance for credit losses on loans		$ 3,584,000