Loans - Risk Rating of Loans by Class of Loans (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2022	Mar. 31, 2021	Dec. 31, 2021
Recorded investment in loans
Total	$ 1,874,641		$ 2,012,814
Advantage Loan Program
Recorded investment in loans
Total	156,012		171,185
Outstanding principal balance of loans repurchased	0	$ 87,944
Real estate loan, Other
Recorded investment in loans
Total	43,966
Pass
Recorded investment in loans
Total	1,784,428		1,895,663
Special Mention
Recorded investment in loans
Total	22,720		27,761
Substandard
Recorded investment in loans
Total	67,262		83,226
Doubtful
Recorded investment in loans
Total	231		6,164
Residential real estate
Recorded investment in loans
Total	1,580,759		1,704,231
Residential real estate | Real estate loan, first mortgage
Recorded investment in loans
Total	1,567,779		1,690,456
Residential real estate | Real estate loan, second mortgage
Recorded investment in loans
Total	12,980		13,775
Residential real estate | Pass | Real estate loan, first mortgage
Recorded investment in loans
Total	1,529,677		1,644,974
Residential real estate | Pass | Real estate loan, second mortgage
Recorded investment in loans
Total	12,744		13,539
Residential real estate | Substandard | Real estate loan, first mortgage
Recorded investment in loans
Total	37,871		45,249
Residential real estate | Substandard | Real estate loan, second mortgage
Recorded investment in loans
Total	236		236
Residential real estate | Doubtful | Real estate loan, first mortgage
Recorded investment in loans
Total	231		233
Commercial real estate, retail
Recorded investment in loans
Total	219,767		201,240
Commercial real estate, retail | Real estate loan, Retail
Recorded investment in loans
Total	37,737		19,574
Commercial real estate, retail | Real estate loan, Multifamily
Recorded investment in loans
Total	98,672		96,960
Commercial real estate, retail | Real estate loan, Offices
Recorded investment in loans
Total	12,267		12,382
Commercial real estate, retail | Hotels/Single-room occupancy hotels
Recorded investment in loans
Total	15,614		14,221
Commercial real estate, retail | Real estate loan, Industrial
Recorded investment in loans
Total	11,511		7,320
Commercial real estate, retail | Real estate loan, Other
Recorded investment in loans
Total	43,966		50,783
Commercial real estate, retail | Pass | Real estate loan, Retail
Recorded investment in loans
Total	37,737		18,846
Commercial real estate, retail | Pass | Real estate loan, Multifamily
Recorded investment in loans
Total	77,654		75,543
Commercial real estate, retail | Pass | Real estate loan, Offices
Recorded investment in loans
Total	10,318		10,413
Commercial real estate, retail | Pass | Hotels/Single-room occupancy hotels
Recorded investment in loans
Total	8,158		8,205
Commercial real estate, retail | Pass | Real estate loan, Industrial
Recorded investment in loans
Total	11,511		7,320
Commercial real estate, retail | Pass | Real estate loan, Other
Recorded investment in loans
Total	42,418		48,996
Commercial real estate, retail | Special Mention | Real estate loan, Retail
Recorded investment in loans
Total			728
Commercial real estate, retail | Special Mention | Real estate loan, Multifamily
Recorded investment in loans
Total	11,531		8,104
Commercial real estate, retail | Special Mention | Hotels/Single-room occupancy hotels
Recorded investment in loans
Total	5,890
Commercial real estate, retail | Special Mention | Real estate loan, Other
Recorded investment in loans
Total	1,548		1,692
Commercial real estate, retail | Substandard | Real estate loan, Multifamily
Recorded investment in loans
Total	9,487		13,313
Commercial real estate, retail | Substandard | Real estate loan, Offices
Recorded investment in loans
Total	1,949		1,969
Commercial real estate, retail | Substandard | Hotels/Single-room occupancy hotels
Recorded investment in loans
Total	1,566		6,016
Commercial real estate, retail | Substandard | Real estate loan, Other
Recorded investment in loans
Total			95
Construction
Recorded investment in loans
Total	73,778		106,759
Construction | Pass
Recorded investment in loans
Total	53,874		67,254
Construction | Special Mention
Recorded investment in loans
Total	3,751		17,226
Construction | Substandard
Recorded investment in loans
Total	16,153		16,348
Construction | Doubtful
Recorded investment in loans
Total			5,931
Commercial lines of credit
Recorded investment in loans
Total	334		363
Commercial lines of credit | Private banking
Recorded investment in loans
Total	115		116
Commercial lines of credit | C&I lending
Recorded investment in loans
Total	219		247
Commercial lines of credit | Pass | Private banking
Recorded investment in loans
Total	115		116
Commercial lines of credit | Pass | C&I lending
Recorded investment in loans
Total	219		236
Commercial lines of credit | Special Mention | C&I lending
Recorded investment in loans
Total			11
Other consumer
Recorded investment in loans
Total	3		221
Other consumer | Pass
Recorded investment in loans
Total	$ 3		$ 221