Business Segment Information - Operating Statements (Details) - USD ($) $ in Thousands	3 Months Ended				6 Months Ended
	Jun. 30, 2021	Mar. 31, 2021	Jun. 30, 2020	Mar. 31, 2020	Jun. 30, 2021	Jun. 30, 2020	Dec. 31, 2020
Operating Income
Interest income	$ 36,632		$ 37,132		$ 73,393	$ 78,074
Total interest expense	1,756		2,192		3,559	4,554
Net interest income	34,876		34,940		69,834	73,520
Provision for (recapture of) credit losses on loans	(493)		1,114		(2,005)	14,384
Net interest income after provision	35,369		33,826		71,839	59,136
Noninterest income	2,169		2,078		4,470	5,271
Noninterest expense	25,775		21,012		49,019	46,786
Income before income taxes	11,763		14,892		27,290	17,621
Income tax expense	2,950		4,274		7,273	5,142
Net income	8,813	$ 11,204	10,618	$ 1,861	20,017	12,479
Total assets	5,072,875		4,614,401		5,072,875	4,614,401	$ 4,634,114
Loans, net of deferred fees	2,824,796		2,686,389		2,824,796	2,686,389	2,619,261
Goodwill	167,631		167,631		167,631	167,631	167,631
Banking
Operating Income
Interest income	33,860		34,570		67,971	72,635
Intersegment interest allocations	202		218		413	502
Total interest expense	1,756		2,192		3,559	4,554
Net interest income	32,306		32,596		64,825	68,583
Provision for (recapture of) credit losses on loans	(486)		1,107		(1,949)	13,981
Net interest income after provision	32,792		31,489		66,774	54,602
Noninterest income	2,049		1,883		4,209	4,906
Noninterest expense	24,331		19,336		46,228	43,519
Intersegment expense allocations	127		106		233	235
Income before income taxes	10,637		14,142		24,988	16,224
Income tax expense	2,617		4,052		6,593	4,729
Net income	8,020		10,090		18,395	11,495
Total assets	5,003,894		4,546,199		5,003,894	4,546,199
Loans, net of deferred fees	2,777,685		2,644,480		2,777,685	2,644,480
Goodwill	154,587		154,587		154,587	154,587	154,587
Factoring
Operating Income
Interest income	2,772		2,562		5,422	5,439
Intersegment interest allocations	(202)		(218)		(413)	(502)
Net interest income	2,570		2,344		5,009	4,937
Provision for (recapture of) credit losses on loans	(7)		7		(56)	403
Net interest income after provision	2,577		2,337		5,065	4,534
Noninterest income	120		195		261	365
Noninterest expense	1,444		1,676		2,791	3,267
Intersegment expense allocations	(127)		(106)		(233)	(235)
Income before income taxes	1,126		750		2,302	1,397
Income tax expense	333		222		680	413
Net income	793		528		1,622	984
Total assets	68,981		68,202		68,981	68,202
Loans, net of deferred fees	47,111		41,909		47,111	41,909
Goodwill	$ 13,044		$ 13,044		$ 13,044	$ 13,044	$ 13,044