Securities - Schedule of Contractual Maturity of Securities Available for Sale, Excluding Equity Securities (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Amortized Cost
1 year or less	$ 64,728
1 year – 5 years	90,138
5 years – 10 years	65,049
After 10 years	7,519
Total, Amortized Cost	227,434
Total debt securities	548,317	$ 533,203
Fair Value
1 year or less	64,979
1 year – 5 years	93,743
5 years – 10 years	70,592
After 10 years	7,666
Total, Fair Value	236,980
Total debt securities	570,582	542,313
Residential & multi-family mortgage
Amortized Cost
Mortgage & asset backed securities, Amortized Cost	295,695
Total debt securities	295,695	273,404
Fair Value
Mortgage & asset backed securities, Fair Value	307,520
Total debt securities	307,520	276,636
Pooled SBA
Amortized Cost
Mortgage & asset backed securities, Amortized Cost	24,168
Total debt securities	24,168	25,063
Fair Value
Mortgage & asset backed securities, Fair Value	25,101
Total debt securities	25,101	25,174
Other
Amortized Cost
Mortgage & asset backed securities, Amortized Cost	1,020
Total debt securities	1,020	1,020
Fair Value
Mortgage & asset backed securities, Fair Value	981
Total debt securities	$ 981	$ 964