Loans Held for Investment (Details) (USD $)					3 Months Ended
	Mar. 31, 2013	Dec. 31, 2012	Mar. 31, 2012	Dec. 31, 2011	Mar. 31, 2013 Maximum	Mar. 31, 2013 Canyon National	Mar. 31, 2013 Palm Desert National	Mar. 31, 2013 FAB	Mar. 31, 2013 Business loans: Commercial and industrial	Dec. 31, 2012 Business loans: Commercial and industrial	Mar. 31, 2012 Business loans: Commercial and industrial	Dec. 31, 2011 Business loans: Commercial and industrial	Mar. 31, 2013 Business loans: Commercial owner occupied	Dec. 31, 2012 Business loans: Commercial owner occupied	Mar. 31, 2012 Business loans: Commercial owner occupied	Dec. 31, 2011 Business loans: Commercial owner occupied	Mar. 31, 2013 Business loans: SBA	Dec. 31, 2012 Business loans: SBA	Mar. 31, 2012 Business loans: SBA	Dec. 31, 2011 Business loans: SBA	Mar. 31, 2013 Business loans: Warehouse facilities	Dec. 31, 2012 Business loans: Warehouse facilities	Mar. 31, 2012 Business loans: Warehouse facilities	Dec. 31, 2011 Business loans: Warehouse facilities	Mar. 31, 2013 Real estate loans: Commercial non-owner occupied	Dec. 31, 2012 Real estate loans: Commercial non-owner occupied	Mar. 31, 2012 Real estate loans: Commercial non-owner occupied	Dec. 31, 2011 Real estate loans: Commercial non-owner occupied	Mar. 31, 2013 Real estate loans: Multi-family	Dec. 31, 2012 Real estate loans: Multi-family	Mar. 31, 2012 Real estate loans: Multi-family	Dec. 31, 2011 Real estate loans: Multi-family	Mar. 31, 2013 Real estate loans: One-to-four family	Dec. 31, 2012 Real estate loans: One-to-four family	Mar. 31, 2012 Real estate loans: One-to-four family	Dec. 31, 2011 Real estate loans: One-to-four family	Mar. 31, 2013 Real estate loans: Land	Dec. 31, 2012 Real estate loans: Land	Mar. 31, 2012 Real estate loans: Land	Dec. 31, 2011 Real estate loans: Land	Mar. 31, 2013 Other loans	Dec. 31, 2012 Other loans	Mar. 31, 2012 Other loans	Dec. 31, 2011 Other loans
Loans Held for Investment
Total gross loans	$ 945,991,000	$ 986,194,000	$ 695,749,000						$ 140,592,000	$ 115,354,000	$ 83,947,000		$ 166,571,000	$ 150,934,000	$ 146,904,000		$ 5,116,000	$ 6,882,000	$ 3,948,000		$ 138,935,000	$ 195,761,000	$ 44,246,000		$ 256,015,000	$ 253,409,000	$ 168,672,000		$ 139,100,000	$ 156,424,000	$ 185,367,000		$ 87,109,000	$ 97,463,000	$ 52,280,000		$ 7,863,000	$ 8,774,000	$ 7,246,000		$ 4,690,000	$ 1,193,000	$ 3,139,000
Less loans held for sale, net	3,643,000	3,681,000	62,000
Total gross loans held for investment	942,348,000	982,513,000	695,687,000
Less: Deferred loan origination costs (fees) and premiums (discounts), net	(520,000)	(306,000)	(492,000)
Less: Allowance for loan losses	(7,994,000)	(7,994,000)	(8,116,000)	(8,522,000)					(2,296,000)	(1,310,000)	(1,462,000)	(1,361,000)	(1,665,000)	(1,512,000)	(1,150,000)	(1,119,000)	(50,000)	(79,000)	(171,000)	(80,000)	(730,000)	(1,544,000)	(771,000)	(1,347,000)	(1,403,000)	(1,459,000)	(1,501,000)	(1,287,000)	(506,000)	(1,145,000)	(2,477,000)	(2,281,000)	(1,174,000)	(862,000)	(563,000)	(931,000)	(121,000)	(31,000)		(39,000)	(49,000)	(52,000)	(21,000)	(77,000)
Loans held for investment, net	933,834,000	974,213,000	687,079,000
Mark-to-market discount						2,700,000	4,700,000	157,000
Secured loans limit to one borrower (as a percent)					25.00%
Unsecured loans limit to one borrower (as a percent)					15.00%
Secured loans limit to one borrower	37,600,000
Unsecured loans limit to one borrower	22,600,000
Aggregate outstanding balance of loans to one borrower of secured credit	$ 22,800,000