Loans - Investment of Originated Loans Held-for-Investment by Loan Type and Credit Quality Indicator (Details) - USD ($) $ in Thousands	Jun. 30, 2019	Dec. 31, 2018
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	$ 2,800,816	$ 2,678,877
Pass
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	2,780,704	2,657,356
Special Mention
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	3,542	3,349
Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	16,570	18,172
Real Estate | Multifamily | Less Than 35% LTV
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	204,081	170,832
Real Estate | Multifamily | Less Than 35% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	204,081	170,832
Real Estate | Multifamily | Less Than 35% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	0	0
Real Estate | Multifamily | Less Than 35% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	0	0
Real Estate | Multifamily | Equal to or Greater than 35% LTV
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	1,828,437	1,763,114
Real Estate | Multifamily | Equal to or Greater than 35% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	1,822,249	1,756,882
Real Estate | Multifamily | Equal to or Greater than 35% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	903	613
Real Estate | Multifamily | Equal to or Greater than 35% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	5,285	5,619
Real Estate | Commercial mortgage | Less Than 35% LTV
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	84,501	79,312
Real Estate | Commercial mortgage | Less Than 35% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	84,114	78,917
Real Estate | Commercial mortgage | Less Than 35% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	387	395
Real Estate | Commercial mortgage | Less Than 35% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	0	0
Real Estate | Commercial mortgage | Equal to or Greater than 35% LTV
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	427,807	420,548
Real Estate | Commercial mortgage | Equal to or Greater than 35% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	417,164	409,155
Real Estate | Commercial mortgage | Equal to or Greater than 35% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	1,114	1,137
Real Estate | Commercial mortgage | Equal to or Greater than 35% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	9,529	10,256
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	55,904	57,065
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	54,093	54,912
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	732	747
Real Estate | One-to-four family residential mortgage | Less Than 60% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	1,079	1,406
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	34,328	35,368
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	33,994	34,808
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	0	0
Real Estate | One-to-four family residential mortgage | Equal to or Greater than 60% LTV | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	334	560
Real Estate | Construction and land
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	33,573	26,613
Real Estate | Construction and land | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	33,573	26,613
Real Estate | Construction and land | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	0	0
Real Estate | Construction and land | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	0	0
Real Estate | Home equity and lines of credit
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	87,389	80,315
Real Estate | Home equity and lines of credit | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	87,107	80,077
Real Estate | Home equity and lines of credit | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	26	27
Real Estate | Home equity and lines of credit | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	256	211
Commercial and Industrial
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	42,768	44,190
Commercial and Industrial | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	42,301	43,640
Commercial and Industrial | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	380	430
Commercial and Industrial | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	87	120
Other
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	2,028	1,520
Other | Pass
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	2,028	1,520
Other | Special Mention
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	0	0
Other | Substandard
Financing Receivable, Recorded Investment [Line Items]
Loans receivable	$ 0	$ 0