Valuation (Significant Unobservable Inputs Rollforward) (Details) - Level 3 - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2020	Jun. 30, 2019
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 1,744,583	$ 1,233,319	$ 1,721,275	$ 1,094,815
Assets, Accreted Discounts/Amortized Premiums	(12,017)	(8,631)	(22,172)	(19,524)
Assets, Realized Gain (Loss)	(15,955)	(258)	(9,138)	(2,591)
Assets, Change In Net Unrealized Gain/(Loss)	33,602	8,221	(59,126)	12,231
Assets, Purchases	126,648	200,759	572,618	496,611
Assets, Sales	142,750	110,689	(421,684)	(274,911)
Assets, Transfers into Level 3	169,618	15,320	130,091	24,384
Assets, Transfers out of Level 3	18,879	13,450	(27,014)	(6,424)
Assets, Ending Balance	1,884,850	1,324,591	1,884,850	1,324,591
Liabilities, Begining Balance	550,507	(282,124)	(594,832)	(297,948)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	(536)	0	(504)	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(719)	(17)	(1,098)	40
Liabilities Purchases/Payments	42,047	25,861	86,761	41,628
Liabilities, Sales/Issuance	233,050	(219,536)	(233,092)	(219,536)
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(742,765)	(475,816)	(742,765)	(475,816)
Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	549,668	(282,124)	(594,396)	(297,948)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	0	0	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(1,481)	(17)	(1,457)	40
Liabilities, Payments	41,553	25,861	86,257	41,628
Liabilities, Issuances	233,092	(219,536)	(233,092)	(219,536)
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(742,688)	(475,816)	(742,688)	(475,816)
Change in unrealized gains (losses), liabilities	(1,500)	(17)	(1,500)	40
Financial Derivatives - Liabilities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gains (losses), liabilities	800		(100)
Securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	9,300	5,700	(87,800)	6,100
Securities | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	20,981	6,389	19,904	7,293
Assets, Accreted Discounts/Amortized Premiums	(1,525)	(676)	(3,245)	(1,449)
Assets, Realized Gain (Loss)	(1)	(450)	(2)	(1,075)
Assets, Change In Net Unrealized Gain/(Loss)	(592)	349	2,163	642
Assets, Purchases	3,448	1,594	5,660	1,600
Assets, Sales	0	236	0	(236)
Assets, Transfers into Level 3	393	4,889	411	5,596
Assets, Transfers out of Level 3	9,797	825	(11,984)	(1,337)
Assets, Ending Balance	12,907	11,034	12,907	11,034
Securities | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	94,197	94,670	89,581	91,291
Assets, Accreted Discounts/Amortized Premiums	413	108	510	195
Assets, Realized Gain (Loss)	444	(384)	406	(650)
Assets, Change In Net Unrealized Gain/(Loss)	3,282	2,909	(7,809)	2,358
Assets, Purchases	26,811	2,050	62,724	17,595
Assets, Sales	10,744	3,294	(27,817)	(20,220)
Assets, Transfers into Level 3	20,959	6,366	15,281	11,308
Assets, Transfers out of Level 3	6,755	5,635	(4,269)	(5,087)
Assets, Ending Balance	128,607	96,790	128,607	96,790
Securities | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	20,276	5,137	29,805	803
Assets, Accreted Discounts/Amortized Premiums	187	25	433	(23)
Assets, Realized Gain (Loss)	(2,253)	50	(867)	0
Assets, Change In Net Unrealized Gain/(Loss)	3,199	363	(8,279)	(7)
Assets, Purchases	2,966	0	40,235	0
Assets, Sales	3,750	374	(32,289)	0
Assets, Transfers into Level 3	47,397	1,077	44,886	5,505
Assets, Transfers out of Level 3	2,327	0	8,229	0
Assets, Ending Balance	65,695	6,278	65,695	6,278
Securities | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	43,804	21,438	44,979	14,915
Assets, Accreted Discounts/Amortized Premiums	(236)	165	(41)	(483)
Assets, Realized Gain (Loss)	(12,850)	(1,151)	(7,518)	(1,676)
Assets, Change In Net Unrealized Gain/(Loss)	4,806	772	(28,099)	1,660
Assets, Purchases	73	0	61,933	831
Assets, Sales	0	0	(1,769)	0
Assets, Transfers into Level 3	100,869	2,988	69,513	1,975
Assets, Transfers out of Level 3	0	6,990	(2,532)	0
Assets, Ending Balance	136,466	17,222	136,466	17,222
Securities | Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	54,627	24,108	48,610	22,800
Assets, Accreted Discounts/Amortized Premiums	(1,104)	(360)	(2,148)	(969)
Assets, Realized Gain (Loss)	(46)	(152)	(196)	(665)
Assets, Change In Net Unrealized Gain/(Loss)	(306)	95	(2,666)	857
Assets, Purchases	1,274	4,610	17,545	9,551
Assets, Sales	6,504	3,282	(13,204)	(6,555)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	47,941	25,019	47,941	25,019
Securities | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	610	5,737	1,113	6,318
Assets, Accreted Discounts/Amortized Premiums	0	6	0	22
Assets, Realized Gain (Loss)	208	(345)	208	(345)
Assets, Change In Net Unrealized Gain/(Loss)	578	(1)	483	(78)
Assets, Purchases	3,411	2,449	3,421	2,832
Assets, Sales	2,856	3,765	(3,274)	(4,668)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	1,951	4,081	1,951	4,081
Securities | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	712	1,469	1,394	1,534
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	7	(142)	7	(142)
Assets, Change In Net Unrealized Gain/(Loss)	319	464	(668)	399
Assets, Purchases	58	0	363	0
Assets, Sales	12	0	(12)	0
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	1,084	1,791	1,084	1,791
Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	16,300	3,800	(14,100)	6,000
Loans | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	939,372	583,252	932,203	496,829
Assets, Accreted Discounts/Amortized Premiums	(1,948)	(1,347)	(2,406)	(2,274)
Assets, Realized Gain (Loss)	(781)	2,089	(576)	1,954
Assets, Change In Net Unrealized Gain/(Loss)	15,660	2,257	(8,663)	4,158
Assets, Purchases	70,163	103,844	201,233	261,446
Assets, Sales	74,019	26,215	(173,344)	(98,233)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	948,447	663,880	948,447	663,880
Loans | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	303,300	239,623	274,759	195,301
Assets, Accreted Discounts/Amortized Premiums	97	834	96	1,139
Assets, Realized Gain (Loss)	(761)	1,413	99	1,414
Assets, Change In Net Unrealized Gain/(Loss)	195	(2,018)	(134)	(2,352)
Assets, Purchases	1,159	47,555	88,727	96,412
Assets, Sales	8,494	27,373	(68,051)	(31,880)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	295,496	260,034	295,496	260,034
Loans | Consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	194,803	192,115	186,954	183,961
Assets, Accreted Discounts/Amortized Premiums	(7,901)	(7,386)	(15,371)	(15,958)
Assets, Realized Gain (Loss)	(11)	(1,455)	15	(3,510)
Assets, Change In Net Unrealized Gain/(Loss)	413	857	(5,338)	2,699
Assets, Purchases	15,218	16,626	76,318	70,882
Assets, Sales	35,841	38,148	(75,897)	(75,465)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	166,681	162,609	166,681	162,609
Loans | Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	6,114	0	18,510	0
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	0	0	0	0
Assets, Change In Net Unrealized Gain/(Loss)	0	0	0	0
Assets, Purchases	113	5,000	217	5,000
Assets, Sales	0	0	(12,500)	0
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	6,227	5,000	6,227	5,000
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	65,397	58,148	71,850	72,298
Assets, Accreted Discounts/Amortized Premiums	0	0	0	276
Assets, Realized Gain (Loss)	0	124	0	1,684
Assets, Change In Net Unrealized Gain/(Loss)	5,643	2,230	(854)	2,190
Assets, Purchases	1,933	17,026	14,216	30,454
Assets, Sales	420	7,852	(12,659)	(37,226)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	72,553	69,676	72,553	69,676
Change in unrealized gain (loss), assets	5,500	2,000	(1,100)	(200)
Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	400	(56)	900	(300)
Financial Derivatives - Assets | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	37	0	620	0
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	51	17	242	17
Assets, Change In Net Unrealized Gain/(Loss)	405	87	(178)	87
Assets, Purchases	0	0	0	0
Assets, Sales	51	17	(242)	(17)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	442	87	442	87
Financial Derivatives - Assets | Credit default swaps on asset-backed securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	353	1,233	993	1,472
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	38	128	(956)	403
Assets, Change In Net Unrealized Gain/(Loss)	0	(143)	916	(382)
Assets, Purchases	21	5	26	8
Assets, Sales	59	133	(626)	(411)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	353	$ 1,090	353	$ 1,090
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	839		(436)
Liabilities, Accreted Discounts/Amortized Premiums	0		0
Liabilities, Realized Gain/(Loss)	(536)		(504)
Liabilities, Change In Net Unrealized Gain/(Loss)	762		359
Liabilities, Payments	494		504
Liabilities, Issuances	(42)		0
Liabilities, Transfers into Level 3	0		0
Liabilities, Transfers out of Level 3	0		0
Liabilities, Ending Balance	$ (77)		$ (77)