Loans - Investment in Nonaccrual and Loans Past Due Still Accruing by Class of Loans (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019
Investment in nonaccrual and loans past due still accruing by class of loans
Nonaccrual	$ 86,424	$ 14,732
Loans Past Due Over 90 Days Still Accruing	46	50
Total Past Due	155,261	78,260
Loans Not Past Due	2,351,482	2,835,000
Total Loans	2,506,743	2,913,260
30 - 59 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	51,619	58,071
60 - 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	17,172	5,407
Greater than 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	86,470	14,782
Residential real estate
Investment in nonaccrual and loans past due still accruing by class of loans
Total Loans	2,033,526	2,476,866
Residential real estate \| Real estate loan, first mortgage
Investment in nonaccrual and loans past due still accruing by class of loans
Nonaccrual	20,043	14,482
Loans Past Due Over 90 Days Still Accruing	46	50
Total Past Due	72,432	55,756
Loans Not Past Due	1,943,602	2,396,800
Total Loans	2,016,034	2,452,556
Residential real estate \| Real estate loan, first mortgage \| 30 - 59 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	37,819	36,112
Residential real estate \| Real estate loan, first mortgage \| 60 - 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	14,524	5,112
Residential real estate \| Real estate loan, first mortgage \| Greater than 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	20,089	14,532
Residential real estate \| Real estate loan, second mortgage
Investment in nonaccrual and loans past due still accruing by class of loans
Nonaccrual	686	210
Total Past Due	1,182	602
Loans Not Past Due	16,310	23,708
Total Loans	17,492	24,310
Residential real estate \| Real estate loan, second mortgage \| 30 - 59 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	362	97
Residential real estate \| Real estate loan, second mortgage \| 60 - 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	134	295
Residential real estate \| Real estate loan, second mortgage \| Greater than 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	686	210
Commercial real estate
Investment in nonaccrual and loans past due still accruing by class of loans
Total Loans	259,958	240,081
Commercial real estate \| Real estate loan, Retail
Investment in nonaccrual and loans past due still accruing by class of loans
Nonaccrual	20	40
Total Past Due	1,030	40
Loans Not Past Due	15,170	6,089
Total Loans	16,200	6,129
Commercial real estate \| Real estate loan, Retail \| 30 - 59 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	1,010
Commercial real estate \| Real estate loan, Retail \| Greater than 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	20	40
Commercial real estate \| Real estate loan, Multifamily
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	3,835
Loans Not Past Due	75,374	64,873
Total Loans	79,209	64,873
Commercial real estate \| Real estate loan, Multifamily \| 30 - 59 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	3,835
Commercial real estate \| Real estate loan, Offices
Investment in nonaccrual and loans past due still accruing by class of loans
Loans Not Past Due	27,061	28,048
Total Loans	27,061	28,048
Commercial real estate \| Real estate loan, Industrial
Investment in nonaccrual and loans past due still accruing by class of loans
Loans Not Past Due	13,186	14,150
Total Loans	13,186	14,150
Commercial real estate \| Real estate loan, Other
Investment in nonaccrual and loans past due still accruing by class of loans
Loans Not Past Due	56,667	45,111
Total Loans	56,667	45,111
Commercial real estate \| Hotels/SROs
Investment in nonaccrual and loans past due still accruing by class of loans
Nonaccrual	19,945
Total Past Due	19,945	5,605
Loans Not Past Due	47,690	76,165
Total Loans	67,635	81,770
Commercial real estate \| Hotels/SROs \| 30 - 59 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due		5,605
Commercial real estate \| Hotels/SROs \| Greater than 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	19,945
Construction
Investment in nonaccrual and loans past due still accruing by class of loans
Nonaccrual	41,873
Total Past Due	52,980	15,008
Loans Not Past Due	153,601	163,368
Total Loans	206,581	178,376
Construction \| 30 - 59 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	8,593	15,008
Construction \| 60 - 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	2,514
Construction \| Greater than 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	41,873
Commercial lines of credit
Investment in nonaccrual and loans past due still accruing by class of loans
Total Loans	6,671	17,903
Commercial lines of credit \| Private banking
Investment in nonaccrual and loans past due still accruing by class of loans
Nonaccrual	2,285
Total Past Due	2,285
Loans Not Past Due	124	11,914
Total Loans	2,409	11,914
Commercial lines of credit \| Private banking \| Greater than 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	2,285
Commercial lines of credit \| C&I lending
Investment in nonaccrual and loans past due still accruing by class of loans
Nonaccrual	1,572
Total Past Due	1,572	1,249
Loans Not Past Due	2,690	4,740
Total Loans	4,262	5,989
Commercial lines of credit \| C&I lending \| 30 - 59 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due		1,249
Commercial lines of credit \| C&I lending \| Greater than 89 Days Past Due
Investment in nonaccrual and loans past due still accruing by class of loans
Total Past Due	1,572
Other consumer
Investment in nonaccrual and loans past due still accruing by class of loans
Loans Not Past Due	7	34
Total Loans	$ 7	$ 34

Amount of financing receivable 90 days or more past due and still accruing.

Reference 1: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 326
-SubTopic 20
-Section 50
-Paragraph 16
-Subparagraph (c)
-URI http://asc.fasb.org/extlink&oid=121599337&loc=SL82919258-210447

Reference 2: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 10
-Section 50
-Paragraph 7
-Subparagraph (d)
-URI http://asc.fasb.org/extlink&oid=121611835&loc=d3e5111-111524

Reference 3: http://www.xbrl.org/2003/role/exampleRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 326
-SubTopic 20
-Section 55
-Paragraph 80
-URI http://asc.fasb.org/extlink&oid=121647567&loc=SL82922355-210448

+ Details

Name:	us-gaap_FinancingReceivableRecordedInvestment90DaysPastDueAndStillAccruing
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Amount of financing receivable not past due.

+ References

Reference 1: http://fasb.org/us-gaap/role/ref/otherTransitionRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 10
-Section 50
-Paragraph 7A
-URI http://asc.fasb.org/extlink&oid=121611835&loc=SL6953401-111524

Reference 2: http://www.xbrl.org/2003/role/exampleRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 326
-SubTopic 20
-Section 55
-Paragraph 80
-URI http://asc.fasb.org/extlink&oid=121647567&loc=SL82922355-210448

Reference 3: http://www.xbrl.org/2009/role/commonPracticeRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 326
-SubTopic 20
-Section 50
-Paragraph 14
-URI http://asc.fasb.org/extlink&oid=121599337&loc=SL82919253-210447

+ Details

Name:	us-gaap_FinancingReceivableRecordedInvestmentCurrent
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Amount of financing receivable on nonaccrual status.

+ References

Reference 1: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 10
-Section 50
-Paragraph 7
-Subparagraph (a)
-URI http://asc.fasb.org/extlink&oid=121611835&loc=d3e5111-111524

Reference 2: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 326
-SubTopic 20
-Section 50
-Paragraph 16
-Subparagraph (a)
-URI http://asc.fasb.org/extlink&oid=121599337&loc=SL82919258-210447

+ Details

Name:	us-gaap_FinancingReceivableRecordedInvestmentNonaccrualStatus
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Amount of financing receivable past due.

+ References

Reference 1: http://www.xbrl.org/2003/role/exampleRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 326
-SubTopic 20
-Section 55
-Paragraph 80
-URI http://asc.fasb.org/extlink&oid=121647567&loc=SL82922355-210448

Reference 2: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 326
-SubTopic 20
-Section 50
-Paragraph 14
-URI http://asc.fasb.org/extlink&oid=121599337&loc=SL82919253-210447

Reference 3: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 10
-Section 50
-Paragraph 7A
-URI http://asc.fasb.org/extlink&oid=121611835&loc=SL6953401-111524

+ Details

Name:	us-gaap_FinancingReceivableRecordedInvestmentPastDue
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Line items represent financial concepts included in a table. These concepts are used to disclose reportable information associated with domain members defined in one or many axes to the table.

+ References

No definition available.

+ Details

Name:	us-gaap_FinancingReceivableRecordedInvestmentPastDueLineItems
Namespace Prefix:	us-gaap_
Data Type:	xbrli:stringItemType
Balance Type:	na
Period Type:	duration

- Definition

Amount before allowance and after deduction of deferred interest and fees, unamortized costs and premiums and discounts from face amounts, of loans and leases held in portfolio, including but not limited to, commercial and consumer loans. Excludes loans and leases covered under loss sharing agreements.

+ References

Reference 1: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 10
-Section 45
-Paragraph 2
-URI http://asc.fasb.org/extlink&oid=121593590&loc=d3e4428-111522

Reference 2: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 310
-SubTopic 10
-Section 50
-Paragraph 4
-URI http://asc.fasb.org/extlink&oid=121611835&loc=d3e5074-111524

+ Details

Name:	us-gaap_LoansAndLeasesReceivableNetOfDeferredIncome
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant