Income Taxes - Schedule of Deferred Tax Assets and Liabilities (Details) - USD ($) $ in Thousands	Jun. 30, 2019	Jun. 30, 2018
Deferred tax assets:
Alternative minimum tax credit	$ 4,799	$ 4,920
Allowance for loan losses	4,685	4,637
Deferred compensation and post-retirement benefits	8,988	9,400
Accrued vacation and sick leave	18	18
Impairments on real estate owned	461	495
Other than temporary impairment on investments	2,232	2,254
Net operating loss carryforward	5,092	8,635
Discount from business combination	2,373	2,605
Unrealized loss on securities held for sale	0	477
Stock compensation plans	2,162	2,271
Other	1,140	1,562
Total gross deferred tax assets	31,950	37,274
Less valuation allowance	0	(325)
Deferred tax assets	31,950	36,949
Deferred tax (liabilities):
Depreciable basis of fixed assets	(1,089)	(566)
Deferred loan fees	(520)	(453)
FHLB stock, book basis in excess of tax	(89)	(89)
Unrealized gain on securities available for sale	(219)	0
Other	(3,510)	(3,276)
Total gross deferred tax liabilities	(5,427)	(4,384)
Net deferred tax assets	$ 26,523	$ 32,565