Valuation (Fair Value Measurement Using Significant Unobservable Inputs) (Details) - Level 3 [Member] - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2018	Jun. 30, 2017	Jun. 30, 2018	Jun. 30, 2017
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 758,723	$ 629,362	$ 661,588	$ 523,837
Assets, Accreted Discounts/Amortized Premiums	(9,827)	(9,785)	(14,251)	(13,979)
Assets, Realized Gain (Loss)	3,216	2,115	7,944	2,858
Assets, Change In Net Unrealized Gain/(Loss)	1,848	5,455	11,688	15,656
Assets, Purchases	223,767	156,110	386,854	353,763
Assets, Sales	(112,690)	(169,558)	(202,131)	(275,956)
Assets, Transfers into Level 3	3,073	16,089	9,301	11,348
Assets, Transfers out of Level 3	(51,604)	(92,443)	(44,487)	(80,182)
Assets, Ending Balance	816,506	537,345	816,506	537,345
Liabilities, Begining Balance	(113,775)	(218)	(125,105)	(505)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	0	(104)	0	(757)
Liabilities, Change In Net Unrealized Gain/(Loss)	414	12	1,198	727
Liabilities Purchases/Payments	12,261	467	22,807	769
Liabilities, Sales/Issuance	0	(364)	0	(441)
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(101,100)	(207)	(101,100)	(207)
Other secured borrowings [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	(113,775)		(125,105)
Liabilities, Accreted Discounts/Amortized Premiums	0		0
Liabilities, Realized Gain/(Loss)	0		0
Liabilities, Change In Net Unrealized Gain/(Loss)	414		1,198
Liabilities, Payments	12,261		22,807
Liabilities, Issuances	0		0
Liabilities, Transfers into Level 3	0		0
Liabilities, Transfers out of Level 3	0		0
Liabilities, Ending Balance	(101,100)		(101,100)
Fair Value, Liabilities Measured on Recurring Basis, Change in Unrealized Gain (Loss)	400		1,200
Financial Derivatives - Liabilities [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Fair Value, Liabilities Measured on Recurring Basis, Change in Unrealized Gain (Loss)		10		50
Total Return Swaps [Member] | Financial Derivatives - Liabilities [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance		0		(249)
Liabilities, Accreted Discounts/Amortized Premiums		0		0
Liabilities, Realized Gain/(Loss)		(85)		(292)
Liabilities, Change In Net Unrealized Gain/(Loss)		0		250
Liabilities, Purchases		21		304
Liabilities, Sales		64		(13)
Liabilities, Transfers into Level 3		0		0
Liabilities, Transfers out of Level 3		0		0
Liabilities, Ending Balance		0		0
Credit Default Swaps On Asset Backed Securities [Member] | Financial Derivatives - Liabilities [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance		(218)		(256)
Liabilities, Accreted Discounts/Amortized Premiums		0		0
Liabilities, Realized Gain/(Loss)		(19)		(465)
Liabilities, Change In Net Unrealized Gain/(Loss)		12		477
Liabilities, Purchases		446		465
Liabilities, Sales		(428)		(428)
Liabilities, Transfers into Level 3		0		0
Liabilities, Transfers out of Level 3		0		0
Liabilities, Ending Balance		(207)		(207)
Investments [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	755,654	623,534	658,448	518,250
Assets, Accreted Discounts/Amortized Premiums	(9,827)	(9,785)	(14,251)	(13,979)
Assets, Realized Gain (Loss)	3,055	1,719	7,697	2,368
Assets, Change In Net Unrealized Gain/(Loss)	2,326	6,176	12,237	16,035
Assets, Purchases	223,745	156,100	386,809	353,695
Assets, Sales	(112,507)	(169,152)	(201,839)	(275,297)
Assets, Transfers into Level 3	3,073	16,089	9,301	11,348
Assets, Transfers out of Level 3	(51,604)	(92,443)	(44,487)	(80,182)
Assets, Ending Balance	813,915	532,238	813,915	532,238
Fair Value, Assets Measured on Recurring Basis, Change in Unrealized Gain (Loss)	1,600	5,100	10,500	9,000
Investments [Member] | Agency residential mortgage-backed securities [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	6,128	29,425	6,173	29,622
Assets, Accreted Discounts/Amortized Premiums	(547)	(2,320)	(1,172)	(4,913)
Assets, Realized Gain (Loss)	18	(402)	57	(437)
Assets, Change In Net Unrealized Gain/(Loss)	(41)	(163)	229	65
Assets, Purchases	1,533	(417)	2,635	1,785
Assets, Sales	(569)	(126)	(957)	(125)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	(633)	(21,101)	(1,076)	(21,101)
Assets, Ending Balance	5,889	4,896	5,889	4,896
Investments [Member] | Private Label Residential Mortgage-Backed Securities [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	111,862	80,332	101,297	90,083
Assets, Accreted Discounts/Amortized Premiums	(1,889)	(476)	156	1,036
Assets, Realized Gain (Loss)	211	1,137	2,719	434
Assets, Change In Net Unrealized Gain/(Loss)	(1,845)	3,327	(986)	6,034
Assets, Purchases	13,596	27,972	24,623	26,800
Assets, Sales	(10,999)	(9,733)	(29,439)	(29,349)
Assets, Transfers into Level 3	3,073	16,089	9,301	11,348
Assets, Transfers out of Level 3	(17,613)	(39,525)	(11,275)	(27,263)
Assets, Ending Balance	96,396	79,123	96,396	79,123
Investments [Member] | Private Label Commercial Mortgage-Backed Securities [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	13,709	41,300	12,347	43,268
Assets, Accreted Discounts/Amortized Premiums	28	276	(163)	632
Assets, Realized Gain (Loss)	(104)	(3,338)	1,618	(2,996)
Assets, Change In Net Unrealized Gain/(Loss)	784	4,991	1,043	6,402
Assets, Purchases	0	20	8,046	20
Assets, Sales	(4,194)	(16,734)	(13,890)	(20,811)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	(1,462)	(12,706)	(240)	(12,706)
Assets, Ending Balance	8,761	13,809	8,761	13,809
Investments [Member] | Commercial Mortgage Loans [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	109,294	62,508	108,301	61,129
Assets, Accreted Discounts/Amortized Premiums	226	101	844	631
Assets, Realized Gain (Loss)	553	79	884	416
Assets, Change In Net Unrealized Gain/(Loss)	610	(56)	449	1,150
Assets, Purchases	20,132	4,500	24,119	27,545
Assets, Sales	(25,864)	(1,236)	(29,646)	(24,975)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	104,951	65,896	104,951	65,896
Investments [Member] | Residential Mortgage Loans [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	240,781	112,650	182,472	84,290
Assets, Accreted Discounts/Amortized Premiums	(464)	537	(1,179)	678
Assets, Realized Gain (Loss)	637	1,133	583	1,081
Assets, Change In Net Unrealized Gain/(Loss)	(267)	(481)	(920)	(77)
Assets, Purchases	76,785	37,048	149,825	71,186
Assets, Sales	(24,000)	(14,790)	(37,309)	(21,061)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	293,472	136,097	293,472	136,097
Investments [Member] | Collateralized Loan Obligations [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	27,479	70,561	24,911	44,956
Assets, Accreted Discounts/Amortized Premiums	(377)	(4,849)	78	(6,032)
Assets, Realized Gain (Loss)	2	532	3	1,453
Assets, Change In Net Unrealized Gain/(Loss)	300	479	527	3,011
Assets, Purchases	11,442	18,157	21,537	56,869
Assets, Sales	(7,889)	(23,233)	(16,099)	(38,609)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	(24,848)	(19,111)	(24,848)	(19,112)
Assets, Ending Balance	6,109	42,536	6,109	42,536
Investments [Member] | Consumer loans and asset-backed securities backed by consumer loans [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	148,422	107,842	135,258	107,157
Assets, Accreted Discounts/Amortized Premiums	(6,955)	(3,208)	(12,851)	(6,264)
Assets, Realized Gain (Loss)	450	478	950	(74)
Assets, Change In Net Unrealized Gain/(Loss)	3,782	(555)	7,585	281
Assets, Purchases	83,069	25,594	125,203	50,012
Assets, Sales	(29,514)	(21,480)	(56,891)	(42,441)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	199,254	108,671	199,254	108,671
Investments [Member] | Corporate Debt [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	18,000	59,609	23,947	25,004
Assets, Accreted Discounts/Amortized Premiums	59	154	(56)	253
Assets, Realized Gain (Loss)	96	216	148	548
Assets, Change In Net Unrealized Gain/(Loss)	(761)	29	(396)	187
Assets, Purchases	448	36,397	905	83,492
Assets, Sales	(1,944)	(75,870)	(8,650)	(88,949)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	(7,048)	0	(7,048)	0
Assets, Ending Balance	8,850	20,535	8,850	20,535
Investments [Member] | Secured Notes [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	0		0
Assets, Accreted Discounts/Amortized Premiums	92		92
Assets, Realized Gain (Loss)	0		0
Assets, Change In Net Unrealized Gain/(Loss)	(234)		(234)
Assets, Purchases	11,268		11,268
Assets, Sales	0		0
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	11,126		11,126
Investments [Member] | Real Estate Owned [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	29,110	25,390	26,277	3,349
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	10	365	(447)	424
Assets, Change In Net Unrealized Gain/(Loss)	(1)	(401)	615	(295)
Assets, Purchases	5,472	54	9,570	24,211
Assets, Sales	(252)	(431)	(1,676)	(2,712)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	34,339	24,977	34,339	24,977
Investments [Member] | Corporate Equity Investments [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	50,869	33,917	37,465	29,392
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	1,182	1,519	1,182	1,519
Assets, Change In Net Unrealized Gain/(Loss)	(1)	(994)	4,325	(723)
Assets, Purchases	0	6,775	9,078	11,775
Assets, Sales	(7,282)	(5,519)	(7,282)	(6,265)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	44,768	35,698	44,768	35,698
Financial Derivatives - Assets [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	3,069	5,828	3,140	5,587
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	161	396	247	490
Assets, Change In Net Unrealized Gain/(Loss)	(478)	(721)	(549)	(379)
Assets, Purchases	22	10	45	68
Assets, Sales	(183)	(406)	(292)	(659)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	2,591	5,107	2,591	5,107
Fair Value, Assets Measured on Recurring Basis, Change in Unrealized Gain (Loss)	(500)	(700)	(500)	(200)
Financial Derivatives - Assets [Member] | Total Return Swaps [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		0		155
Assets, Accreted Discounts/Amortized Premiums		0		0
Assets, Realized Gain (Loss)		65		222
Assets, Change In Net Unrealized Gain/(Loss)		0		(155)
Assets, Purchases		0		0
Assets, Sales		(65)		(222)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		0		0
Financial Derivatives - Assets [Member] | Warrant [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance				106
Assets, Accreted Discounts/Amortized Premiums				0
Assets, Realized Gain (Loss)				(100)
Assets, Change In Net Unrealized Gain/(Loss)				(6)
Assets, Purchases				0
Assets, Sales				0
Assets, Transfers into Level 3				0
Assets, Transfers out of Level 3				0
Assets, Ending Balance		0		0
Financial Derivatives - Assets [Member] | Credit Default Swaps On Asset Backed Securities [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	3,069	5,828	3,140	5,326
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	161	331	247	368
Assets, Change In Net Unrealized Gain/(Loss)	(478)	(721)	(549)	(218)
Assets, Purchases	22	10	45	68
Assets, Sales	(183)	(341)	(292)	(437)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	$ 2,591	$ 5,107	$ 2,591	$ 5,107