Loans - Schedule of Debt Restructurings on Financing Receivables (Details) $ in Thousands	12 Months Ended
	Jun. 30, 2019 USD ($) loan	Jun. 30, 2018 USD ($) loan	Jun. 30, 2017 USD ($) loan
Financing Receivable, Modifications
Number of loans | loan	29	31	41
Pre-modification outstanding recorded investment	$ 7,291	$ 4,198	$ 4,915
Post-modification outstanding recorded investment	$ 7,249	$ 4,283	$ 4,472
Extended term
Financing Receivable, Modifications
Number of loans | loan	3	5	12
Pre-modification outstanding recorded investment	$ 68	$ 550	$ 1,005
Post-modification outstanding recorded investment	$ 67	$ 534	$ 980
Other Troubled Debt Restructing [Member]
Financing Receivable, Modifications
Number of loans | loan	25	26	24
Pre-modification outstanding recorded investment	$ 7,138	$ 3,648	$ 3,542
Post-modification outstanding recorded investment	$ 7,098	$ 3,749	$ 3,128
Retail Consumer Loans | Below market interest rate
Financing Receivable, Modifications
Number of loans | loan	1	0	5
Pre-modification outstanding recorded investment	$ 85	$ 0	$ 368
Post-modification outstanding recorded investment	$ 84	$ 0	$ 364
Retail Consumer Loans | Below market interest rate | One-to-four family
Financing Receivable, Modifications
Number of loans | loan	1	0	3
Pre-modification outstanding recorded investment	$ 85	$ 0	$ 288
Post-modification outstanding recorded investment	$ 84	$ 0	$ 285
Retail Consumer Loans | Below market interest rate | Construction and land/lots
Financing Receivable, Modifications
Number of loans | loan	0	0	2
Pre-modification outstanding recorded investment	$ 0	$ 0	$ 80
Post-modification outstanding recorded investment	$ 0	$ 0	$ 79
Retail Consumer Loans | Extended term | One-to-four family
Financing Receivable, Modifications
Number of loans | loan	1	4	5
Pre-modification outstanding recorded investment	$ 34	$ 514	$ 186
Post-modification outstanding recorded investment	$ 34	$ 502	$ 179
Retail Consumer Loans | Extended term | HELOCs - originated
Financing Receivable, Modifications
Number of loans | loan	0	0	1
Pre-modification outstanding recorded investment	$ 0	$ 0	$ 37
Post-modification outstanding recorded investment	$ 0	$ 0	$ 37
Retail Consumer Loans | Extended term | Construction and land/lots
Financing Receivable, Modifications
Number of loans | loan	0	1	1
Pre-modification outstanding recorded investment	$ 0	$ 36	$ 280
Post-modification outstanding recorded investment	$ 0	$ 32	$ 264
Retail Consumer Loans | Extended term | Consumer
Financing Receivable, Modifications
Number of loans | loan	2	0	2
Pre-modification outstanding recorded investment	$ 34	$ 0	$ 11
Post-modification outstanding recorded investment	$ 33	$ 0	$ 11
Retail Consumer Loans | Other Troubled Debt Restructing [Member] | One-to-four family
Financing Receivable, Modifications
Number of loans | loan	18	25	13
Pre-modification outstanding recorded investment	$ 1,452	$ 3,646	$ 525
Post-modification outstanding recorded investment	$ 1,433	$ 3,747	$ 517
Retail Consumer Loans | Other Troubled Debt Restructing [Member] | HELOCs - originated
Financing Receivable, Modifications
Number of loans | loan	0	0	2
Pre-modification outstanding recorded investment	$ 0	$ 0	$ 33
Post-modification outstanding recorded investment	$ 0	$ 0	$ 31
Retail Consumer Loans | Other Troubled Debt Restructing [Member] | Construction and land/lots
Financing Receivable, Modifications
Number of loans | loan	1	0	4
Pre-modification outstanding recorded investment	$ 29	$ 0	$ 404
Post-modification outstanding recorded investment	$ 28	$ 0	$ 318
Retail Consumer Loans | Other Troubled Debt Restructing [Member] | Indirect Auto Finance [Member]
Financing Receivable, Modifications
Number of loans | loan	1	0	0
Pre-modification outstanding recorded investment	$ 33	$ 0	$ 0
Post-modification outstanding recorded investment	$ 26	$ 0	$ 0
Retail Consumer Loans | Other Troubled Debt Restructing [Member] | Consumer
Financing Receivable, Modifications
Number of loans | loan	1	1	0
Pre-modification outstanding recorded investment	$ 2	$ 2	$ 0
Post-modification outstanding recorded investment	$ 2	$ 2	$ 0
Commercial Loans | Extended term | Construction and development
Financing Receivable, Modifications
Number of loans | loan	0	0	1
Pre-modification outstanding recorded investment	$ 0	$ 0	$ 439
Post-modification outstanding recorded investment	$ 0	$ 0	$ 439
Commercial Loans | Extended term | Commercial and industrial
Financing Receivable, Modifications
Number of loans | loan	0	0	2
Pre-modification outstanding recorded investment	$ 0	$ 0	$ 52
Post-modification outstanding recorded investment	$ 0	$ 0	$ 50
Commercial Loans | Other Troubled Debt Restructing [Member] | Construction and development
Financing Receivable, Modifications
Number of loans | loan	1	0	0
Pre-modification outstanding recorded investment	$ 182	$ 0	$ 0
Post-modification outstanding recorded investment	$ 182	$ 0	$ 0
Commercial Loans | Other Troubled Debt Restructing [Member] | Commercial real estate
Financing Receivable, Modifications
Number of loans | loan	3	0	3
Pre-modification outstanding recorded investment	$ 5,440	$ 0	$ 2,349
Post-modification outstanding recorded investment	$ 5,427	$ 0	$ 2,035
Commercial Loans | Other Troubled Debt Restructing [Member] | Commercial and industrial
Financing Receivable, Modifications
Number of loans | loan	0	0	2
Pre-modification outstanding recorded investment	$ 0	$ 0	$ 231
Post-modification outstanding recorded investment	$ 0	$ 0	$ 227