FDIC Loss Sharing Asset FDIC Loss Sharing Asset (Details1) (USD $) In Thousands, unless otherwise specified	9 Months Ended
	Sep. 30, 2014	Dec. 31, 2013
Schedule of FDIC Loss Sharing Asset [Line Items]
FDIC loss sharing asset	$ 22,977	$ 45,524
FDIC Indemnification True Up	6,924
NonSingle Family Covered Assets	279,310
Single Family Covered Assets	107,969
Affinity Bank [Member]
Schedule of FDIC Loss Sharing Asset [Line Items]
FDIC loss sharing asset	(819)
NonSingle Family Covered Assets	142,454
Single Family Covered Assets	12,536
Los Padres Bank [Member]
Schedule of FDIC Loss Sharing Asset [Line Items]
FDIC loss sharing asset	13,283
NonSingle Family Covered Assets	92,440
Single Family Covered Assets	64,131
Western Commercial Bank [Member]
Schedule of FDIC Loss Sharing Asset [Line Items]
FDIC loss sharing asset	1,432
FDIC Indemnification True Up	1,574
NonSingle Family Covered Assets	13,746
San Luis Trust Bank [Member]
Schedule of FDIC Loss Sharing Asset [Line Items]
FDIC loss sharing asset	9,081
FDIC Indemnification True Up	5,350
NonSingle Family Covered Assets	30,670
Single Family Covered Assets	$ 31,302