Commitments and Contingencies - Narrative (Details) - USD ($) $ in Thousands	12 Months Ended
	Jun. 30, 2020	Jun. 30, 2019
Loss Contingencies [Line Items]
Undisbursed portions of construction loans	$ 141,557	$ 181,477
Loan commitments	57,798	93,432
Variable rate loan commitments	10,678	34,631
Fixed rate loan commitments	47,120	58,801
Pre-approved but unused lines of credit	398,781	353,663
Cash reserve deposit required and made	0	2,633
Letters of credit outstanding	$ 7,766	$ 9,460
Minimum
Loss Contingencies [Line Items]
Fixed rate interest rate (as a percent)	1.74%	2.69%
Loan commitments terms	3 years
Maximum
Loss Contingencies [Line Items]
Fixed rate interest rate (as a percent)	8.54%	8.59%
Loan commitments terms	30 years