Note 10 - Income Taxes - Deferred Tax Assets and Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Deferred Tax Assets
Loan loss allowance, due to differences in computation of bad debts	$ 38,936	$ 37,567
Share-based compensation	2,945	3,260
Non-accrual interest	1,005	2,277
Write-down on equity securities and venture capital investments	2,010	2,095
Depreciation and amortization	0	3,798
State tax	4,540	4,344
Unrealized loss on interest rate swaps	1,599	190
Unrealized loss on securities available-for-sale, net	0	3,864
Unrealized loss on equity securities	0	843
Tax credits carried forward	9,473	9,473
Net operating loss carried forward	14,247	17,098
Other, net	5,012	4,782
Gross deferred tax assets	79,767	89,591
Deferred Tax Liabilities
Deferred loan costs	(9,778)	(8,839)
Depreciation and amortization	(1,606)	0
Unrealized gain on securities available-for-sale, net	(5,990)	0
Unrealized gain on equity securities	(668)	0
Dividends on Federal Home Loan Bank common stock	(978)	(956)
Other, net	(3,307)	(4,008)
Gross deferred tax liabilities	(22,327)	(13,803)
Net deferred tax assets	$ 57,440	$ 75,788