Loans Receivable Held for Investment (Details 2) (USD $) In Thousands, unless otherwise specified	3 Months Ended														3 Months Ended
	Mar. 31, 2014	Mar. 31, 2013	Mar. 31, 2014 One-to-four units	Mar. 31, 2013 One-to-four units	Mar. 31, 2014 Five or more units	Mar. 31, 2013 Five or more units	Mar. 31, 2014 Commercial real estate	Mar. 31, 2013 Commercial real estate	Mar. 31, 2014 Church	Mar. 31, 2013 Church	Mar. 31, 2014 Construction	Dec. 31, 2013 Construction	Mar. 31, 2013 Construction	Dec. 31, 2012 Construction	Mar. 31, 2014 Commercial	Mar. 31, 2013 Commercial	Mar. 31, 2014 Consumer	Mar. 31, 2013 Consumer	Dec. 31, 2012 Consumer
Allowance for loan losses
Beginning balance	$ 10,146	$ 11,869	$ 1,930	$ 2,060	$ 1,726	$ 2,122	$ 1,473	$ 2,685	$ 4,949	$ 4,818	$ 7	$ 7	$ 8	$ 8	$ 55	$ 167	$ 6	$ 9	$ 9
Provision for (recapture of) loan losses	(1,082)		(55)	(449)	381	(361)	(252)	(207)	(55)	1,029					(1,099)	(12)	(2)
Recoveries	1,240	376	2	259				15	156	7					1,082	95
Loans charged off	(213)	(1,795)	(3)	(36)		(638)	(9)	(944)	(183)	(177)					(18)
Ending balance	$ 10,091	$ 10,450	$ 1,874	$ 1,834	$ 2,107	$ 1,123	$ 1,212	$ 1,549	$ 4,867	$ 5,677	$ 7	$ 7	$ 8	$ 8	$ 20	$ 250	$ 4	$ 9	$ 9