Loans Receivable and Allowance For Loan Losses - Schedule of Aging Analysis of Past Due Loans (Details) - USD ($)	Jun. 30, 2017	Dec. 31, 2016	Jun. 30, 2016
Financing Receivable, Recorded Investment, Past Due [Line Items]
Financing Receivable, Recorded Investment, 90 Days Past Due and Still Accruing	$ 0	$ 0
Total Past Due	1,160,000	1,135,000
Current	719,142,000	604,280,000
Total loans receivable	720,302,000	605,415,000	$ 560,562,000
REAL ESTATE LOANS
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total loans receivable	397,695,000	331,950,000
Total real estate loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	498,000	244,000
Current	397,197,000	331,706,000
Total loans receivable	397,695,000	331,950,000
Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Current	57,997,000	55,871,000
Total loans receivable	57,997,000	55,871,000
Construction and development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Current	119,455,000	94,462,000
Total loans receivable	119,455,000	94,462,000
Home equity
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	354,000	244,000
Current	22,096,000	19,837,000
Total loans receivable	22,450,000	20,081,000
One-to-four-family (1)
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	144,000	0
Current	154,682,000	124,009,000
Total loans receivable	154,826,000	124,009,000
Multi-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Current	42,967,000	37,527,000
Total loans receivable	42,967,000	37,527,000
CONSUMER LOANS
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	662,000	891,000
Current	190,067,000	173,835,000
Total loans receivable	190,729,000	174,726,000	165,736,000
Indirect home improvement
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	516,000	713,000
Current	117,410,000	107,046,000
Total loans receivable	117,926,000	107,759,000
Solar
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	105,000	161,000
Current	38,402,000	36,342,000
Total loans receivable	38,507,000	36,503,000
Marine
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	40,000	8,000
Current	32,214,000	28,541,000
Total loans receivable	32,254,000	28,549,000
Other consumer
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,000	9,000
Current	2,041,000	1,906,000
Total loans receivable	2,042,000	1,915,000
COMMERCIAL BUSINESS LOANS
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Current	131,878,000	98,739,000
Total loans receivable	131,878,000	98,739,000	$ 99,072,000
Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Current	92,713,000	65,841,000
Total loans receivable	92,713,000	65,841,000
Warehouse lending
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Current	39,165,000	32,898,000
Total loans receivable	39,165,000	32,898,000
30-59 Days Past Due
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	361,000	405,000
30-59 Days Past Due | Total real estate loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	34,000
30-59 Days Past Due | Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
30-59 Days Past Due | Construction and development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
30-59 Days Past Due | Home equity
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	34,000
30-59 Days Past Due | One-to-four-family (1)
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
30-59 Days Past Due | Multi-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
30-59 Days Past Due | CONSUMER LOANS
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	361,000	371,000
30-59 Days Past Due | Indirect home improvement
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	272,000	268,000
30-59 Days Past Due | Solar
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	69,000	92,000
30-59 Days Past Due | Marine
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	20,000	8,000
30-59 Days Past Due | Other consumer
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	3,000
30-59 Days Past Due | COMMERCIAL BUSINESS LOANS
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
30-59 Days Past Due | Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
30-59 Days Past Due | Warehouse lending
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	400,000	280,000
60-89 Days Past Due | Total real estate loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	276,000	0
60-89 Days Past Due | Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due | Construction and development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due | Home equity
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	132,000	0
60-89 Days Past Due | One-to-four-family (1)
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	144,000	0
60-89 Days Past Due | Multi-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due | CONSUMER LOANS
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	124,000	280,000
60-89 Days Past Due | Indirect home improvement
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	120,000	278,000
60-89 Days Past Due | Solar
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due | Marine
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	3,000	0
60-89 Days Past Due | Other consumer
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,000	2,000
60-89 Days Past Due | COMMERCIAL BUSINESS LOANS
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due | Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
60-89 Days Past Due | Warehouse lending
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	399,000	450,000
90 Days or More Past Due | Total real estate loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	222,000	210,000
90 Days or More Past Due | Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due | Construction and development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due | Home equity
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	222,000	210,000
90 Days or More Past Due | One-to-four-family (1)
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due | Multi-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due | CONSUMER LOANS
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	177,000	240,000
90 Days or More Past Due | Indirect home improvement
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	124,000	167,000
90 Days or More Past Due | Solar
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	36,000	69,000
90 Days or More Past Due | Marine
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	17,000	0
90 Days or More Past Due | Other consumer
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	4,000
90 Days or More Past Due | COMMERCIAL BUSINESS LOANS
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due | Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
90 Days or More Past Due | Warehouse lending
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Non-Accrual
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	754,000	721,000
Non-Accrual | Total real estate loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	383,000	210,000
Non-Accrual | Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Non-Accrual | Construction and development
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Non-Accrual | Home equity
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due		210,000
Non-Accrual | Home Equity
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	239,000
Non-Accrual | One-to-four-family (1)
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	144,000	0
Non-Accrual | Multi-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Non-Accrual | CONSUMER LOANS
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	371,000	511,000
Non-Accrual | Indirect home improvement
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	307,000	435,000
Non-Accrual | Solar
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	37,000	69,000
Non-Accrual | Marine
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	17,000	0
Non-Accrual | Other consumer
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	10,000	7,000
Non-Accrual | COMMERCIAL BUSINESS LOANS
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Non-Accrual | Commercial and industrial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Non-Accrual | Warehouse lending
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	$ 0	$ 0