FDIC Loss Sharing Asset FDIC Loss Sharing Asset (Details1) (USD $) In Thousands, unless otherwise specified	6 Months Ended
	Jun. 30, 2014		Dec. 31, 2013
Schedule of FDIC Loss Sharing Asset [Line Items]
FDIC Indemnification Asset	$ 28,834		$ 45,524
FDIC Indemnification True Up	6,826
NonSingle Family Covered Assets	319,317	[1]
Single Family Covered Assets	115,098
Affinity Bank [Member]
Schedule of FDIC Loss Sharing Asset [Line Items]
FDIC Indemnification Asset	2,315
NonSingle Family Covered Assets	156,509	[1]
Single Family Covered Assets	12,701
Los Padres Bank [Member]
Schedule of FDIC Loss Sharing Asset [Line Items]
FDIC Indemnification Asset	15,101
NonSingle Family Covered Assets	111,413	[1]
Single Family Covered Assets	69,584
Western Commercial Bank [Member]
Schedule of FDIC Loss Sharing Asset [Line Items]
FDIC Indemnification Asset	1,605
FDIC Indemnification True Up	1,555
NonSingle Family Covered Assets	14,068	[1]
San Luis Trust Bank [Member]
Schedule of FDIC Loss Sharing Asset [Line Items]
FDIC Indemnification Asset	9,813
FDIC Indemnification True Up	5,271
NonSingle Family Covered Assets	37,327	[1]
Single Family Covered Assets	$ 32,813

[1] (1)Excludes securities.