Loans Held for Investment (Details) (USD $)					9 Months Ended
	Sep. 30, 2014	Dec. 31, 2013	Sep. 30, 2013	Dec. 31, 2012	Sep. 30, 2014 Maximum	Sep. 30, 2014 Canyon National	Sep. 30, 2014 Palm Desert National	Sep. 30, 2014 San Diego Trust Bank	Jun. 25, 2013 San Diego Trust Bank	Sep. 30, 2014 FAB	Mar. 15, 2013 FAB	Sep. 30, 2014 Business loans: Commercial and industrial	Dec. 31, 2013 Business loans: Commercial and industrial	Sep. 30, 2013 Business loans: Commercial and industrial	Dec. 31, 2012 Business loans: Commercial and industrial	Sep. 30, 2014 Business loans: Commercial and industrial Canyon National	Sep. 30, 2014 Business loans: Commercial owner occupied	Dec. 31, 2013 Business loans: Commercial owner occupied	Sep. 30, 2013 Business loans: Commercial owner occupied	Dec. 31, 2012 Business loans: Commercial owner occupied	Sep. 30, 2014 Business loans: Commercial owner occupied Canyon National	Sep. 30, 2014 Business loans: SBA	Dec. 31, 2013 Business loans: SBA	Sep. 30, 2013 Business loans: SBA	Dec. 31, 2012 Business loans: SBA	Sep. 30, 2014 Business loans: Warehouse facilities	Dec. 31, 2013 Business loans: Warehouse facilities	Sep. 30, 2013 Business loans: Warehouse facilities	Dec. 31, 2012 Business loans: Warehouse facilities	Sep. 30, 2014 Real estate loans: Commercial non-owner occupied	Dec. 31, 2013 Real estate loans: Commercial non-owner occupied	Sep. 30, 2013 Real estate loans: Commercial non-owner occupied	Dec. 31, 2012 Real estate loans: Commercial non-owner occupied	Sep. 30, 2014 Real estate loans: Commercial non-owner occupied Canyon National	Sep. 30, 2014 Real estate loans: Multi-family	Dec. 31, 2013 Real estate loans: Multi-family	Sep. 30, 2013 Real estate loans: Multi-family	Dec. 31, 2012 Real estate loans: Multi-family	Sep. 30, 2014 Real estate loans: One-to-four family	Dec. 31, 2013 Real estate loans: One-to-four family	Sep. 30, 2013 Real estate loans: One-to-four family	Dec. 31, 2012 Real estate loans: One-to-four family	Sep. 30, 2014 Real estate loans: One-to-four family Palm Desert National	Sep. 30, 2014 Real estate loans: Construction	Dec. 31, 2013 Real estate loans: Construction	Sep. 30, 2013 Real estate loans: Construction	Sep. 30, 2014 Real estate loans: Land	Dec. 31, 2013 Real estate loans: Land	Sep. 30, 2013 Real estate loans: Land	Dec. 31, 2012 Real estate loans: Land	Sep. 30, 2014 Other loans	Dec. 31, 2013 Other loans	Sep. 30, 2013 Other loans	Dec. 31, 2012 Other loans
Loans Held for Investment
Total gross loans	$ 1,547,911,000	$ 1,243,252,000	$ 1,142,015,000									$ 360,700,000	$ 187,035,000	$ 173,720,000			$ 237,996,000	$ 221,089,000	$ 222,162,000			$ 20,482,000	$ 10,659,000	$ 6,455,000		$ 108,093,000	$ 87,517,000	$ 49,104,000		$ 355,984,000	$ 333,544,000	$ 304,979,000			$ 262,588,000	$ 233,689,000	$ 218,929,000		$ 125,326,000	$ 145,235,000	$ 152,667,000			$ 67,118,000	$ 13,040,000	$ 2,835,000	$ 6,103,000	$ 7,605,000	$ 7,371,000		$ 3,521,000	$ 3,839,000	$ 3,793,000
Less loans held for sale, net		3,147,000	3,176,000
Total gross loans held for investment	1,547,911,000	1,240,105,000	1,138,839,000
Deferred loan origination costs / (fees) and premiums / (discounts), net	93,000	18,000	130,000
Allowance for loan losses	(10,767,000)	(8,200,000)	(7,994,000)	(7,994,000)								(3,375,000)	(1,968,000)	(2,801,000)	(1,310,000)		(1,856,000)	(1,818,000)	(1,519,000)	(1,512,000)		(445,000)	(151,000)	(50,000)	(79,000)	(494,000)	(392,000)	(225,000)	(1,544,000)	(1,901,000)	(1,658,000)	(1,403,000)	(1,459,000)		(1,011,000)	(817,000)	(506,000)	(1,145,000)	(815,000)	(1,099,000)	(1,174,000)	(862,000)		(775,000)	(136,000)	(121,000)	(72,000)	(127,000)	(147,000)	(31,000)	(23,000)	(34,000)	(48,000)	(52,000)
Loans held for investment, net	1,537,237,000	1,231,923,000	1,130,975,000
Unaccreted mark-to-market discount						1,500,000	1,700,000	145,000
Mark-to-market premium										31,000
Secured loans limit to one borrower (as a percent)					25.00%
Unsecured loans limit to one borrower (as a percent)					15.00%
Secured loans limit to one borrower	63,200,000
Unsecured loans limit to one borrower	37,900,000
Aggregate outstanding balance of loans to one borrower of secured credit	34,200,000
Purchased Credit Impaired
Total purchase credit impaired	1,626,000					1,618,000	8,000		0		0	92,000				92,000	555,000				555,000									971,000				971,000					8,000				8,000
Purchased credit impaired loans, nonaccrual status	$ 1,600,000