Loans and Allowance for Loan Losses, Allowance for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended				6 Months Ended
	Jun. 30, 2015		Jun. 30, 2014		Jun. 30, 2015	Jun. 30, 2014
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	$ 45,944		$ 47,035		$ 46,327	$ 47,714
Loans charged off [Abstract]
Total loan charge offs	1,318		1,821		2,634	4,138
Recoveries of loans previously charged off [Abstract]
Total recoveries	145		221		278	359
Net loans charged off	1,173		1,600		2,356	3,779
Provision for loan losses	800		1,500		1,600	3,000
Balance at end of period	45,571		46,935		45,571	46,935
Commercial [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	4,024		3,840		4,071	4,019
Loans charged off [Abstract]
Total loan charge offs	50		13		100	886
Recoveries of loans previously charged off [Abstract]
Total recoveries	1		2		18	21
Net loans charged off	49		11		82	865
Provision for loan losses	47		244		33	919
Balance at end of period	4,022		4,073		4,022	4,073
Real Estate Mortgage - 1 to 4 Family [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	41,529		43,091		42,088	43,597
Loans charged off [Abstract]
Total loan charge offs	1,235		1,766		2,458	3,159
Recoveries of loans previously charged off [Abstract]
Total recoveries	135		211		245	325
Net loans charged off	1,100		1,555		2,213	2,834
Provision for loan losses	658		1,216		1,212	1,989
Balance at end of period	41,087		42,752		41,087	42,752
Installment [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	391		104		168	98
Loans charged off [Abstract]
Total loan charge offs	33		42		76	93
Recoveries of loans previously charged off [Abstract]
Total recoveries	9		8		15	13
Net loans charged off	24		34		61	80
Provision for loan losses	95		40		355	92
Balance at end of period	462		110		462	110
New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	1,149	[1]	1,736	[1]	2,356	2,971
Recoveries of loans previously charged off [Abstract]
Total recoveries	142	[1]	203	[1]	274	301
New York and Other States [Member] | Commercial [Member]
Loans charged off [Abstract]
Total loan charge offs	50	[1]	13	[1]	100	273
Recoveries of loans previously charged off [Abstract]
Total recoveries	0	[1]	0	[1]	16	18
New York and Other States [Member] | Real Estate Mortgage - 1 to 4 Family [Member]
Loans charged off [Abstract]
Total loan charge offs	1,066	[1]	1,691	[1]	2,180	2,617
Recoveries of loans previously charged off [Abstract]
Total recoveries	133	[1]	195	[1]	243	270
New York and Other States [Member] | Installment [Member]
Loans charged off [Abstract]
Total loan charge offs	33	[1]	32	[1]	76	81
Recoveries of loans previously charged off [Abstract]
Total recoveries	9	[1]	8	[1]	15	13
Florida [Member]
Loans charged off [Abstract]
Total loan charge offs	169		85		278	1,167
Recoveries of loans previously charged off [Abstract]
Total recoveries	3		18		4	58
Florida [Member] | Commercial [Member]
Loans charged off [Abstract]
Total loan charge offs	0		0		0	613
Recoveries of loans previously charged off [Abstract]
Total recoveries	1		2		2	3
Florida [Member] | Real Estate Mortgage - 1 to 4 Family [Member]
Loans charged off [Abstract]
Total loan charge offs	169		75		278	542
Recoveries of loans previously charged off [Abstract]
Total recoveries	2		16		2	55
Florida [Member] | Installment [Member]
Loans charged off [Abstract]
Total loan charge offs	0		10		0	12
Recoveries of loans previously charged off [Abstract]
Total recoveries	$ 0		$ 0		$ 0	$ 0

[1] Includes New York, New Jersey, Vermont and Massachusetts.