Loans and Allowance for Credit Losses - Balance in the ACL and the recorded investment in loans (Details) - USD ($)	12 Months Ended
	Jun. 30, 2022	Jun. 30, 2021	Jun. 30, 2020
Allowance for credit losses [Abstract]
Balance, beginning of period	$ 33,222,000	$ 25,139,000	$ 19,903,000
Impact of CECL adoption		9,333,000
Provision (benefit) charged to expense	(66,000)	(602,000)	6,002,000
Losses charged off	(153,000)	(734,000)	(853,000)
Recoveries	69,000	86,000	87,000
Negative PCL	(533,000)
Residential Real Estate
Allowance for credit losses [Abstract]
Balance, beginning of period	11,192,000	4,875,000	3,706,000
Impact of CECL adoption		3,521,000
Provision (benefit) charged to expense	(2,238,000)	2,973,000	1,529,000
Losses charged off	(72,000)	(180,000)	(379,000)
Recoveries	3,000	3,000	19,000
Construction Real Estate
Allowance for credit losses [Abstract]
Balance, beginning of period	2,170,000	2,010,000	1,365,000
Impact of CECL adoption		(121,000)
Provision (benefit) charged to expense	46,000	281,000	645,000
Commercial Real Estate
Allowance for credit losses [Abstract]
Balance, beginning of period	14,535,000	12,132,000	9,399,000
Impact of CECL adoption		3,856,000
Provision (benefit) charged to expense	2,251,000	(1,364,000)	2,730,000
Losses charged off		(90,000)	(12,000)
Recoveries		1,000	15,000
Consumer loans
Allowance for credit losses [Abstract]
Balance, beginning of period	916,000	1,182,000	1,046,000
Impact of CECL adoption		1,065,000
Provision (benefit) charged to expense	(205,000)	(1,232,000)	300,000
Losses charged off	(65,000)	(146,000)	(189,000)
Recoveries	64,000	47,000	25,000
Commercial loans
Allowance for credit losses [Abstract]
Balance, beginning of period	4,409,000	4,940,000	4,387,000
Impact of CECL adoption		1,012,000
Provision (benefit) charged to expense	80,000	(1,260,000)	798,000
Losses charged off	(16,000)	(318,000)	(273,000)
Recoveries	$ 2,000	$ 35,000	$ 28,000