Valuation (Significant Unobservable Inputs Rollforward) (Details) - Level 3 - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2021	Jun. 30, 2020	Jun. 30, 2021	Jun. 30, 2020	Dec. 31, 2020
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 1,998,486	$ 1,744,583	$ 1,959,802	$ 1,721,275
Assets, Accreted Discounts/Amortized Premiums	(6,514)	(12,017)	(10,478)	(22,172)
Assets, Realized Gain (Loss)	1,957	(15,955)	3,344	(9,138)
Assets, Change In Net Unrealized Gain/(Loss)	28,673	33,602	47,598	(59,126)
Assets, Purchases	526,832	126,648	891,873	572,618
Assets, Sales	350,732	142,750	(671,899)	(421,684)
Assets, Transfers into Level 3	13,806	169,618	12,005	130,091
Assets, Transfers out of Level 3	(31,396)	18,879	(51,133)	(27,014)
Assets, Ending Balance	2,181,112	1,884,850	2,181,112	1,884,850
Liabilities, Begining Balance	(911,594)	(550,507)	(755,405)	(594,832)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	(860)	(536)	(1,360)	(504)
Liabilities, Change In Net Unrealized Gain/(Loss)	2,287	(719)	3,676	(1,098)
Liabilities Purchases/Payments	231,713	42,047	325,143	86,761
Liabilities, Sales/Issuance	324,639	233,050	(575,147)	(233,092)
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(1,003,093)	(742,765)	(1,003,093)	(742,765)
Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	114,464		114,464		$ 127,838
CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	17,025		17,025		63,148
CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	33,989		33,989		111,100
Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	69,254		69,254		44,925
Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	16,584		16,584
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	178,979		178,979		141,620
Investment in unconsolidated entities | Measurement Input, Net Asset Value | Enterprise Value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	75,820		75,820		$ 62,084
Investment in unconsolidated entities | Measurement Input, Exercise Price | Valuation, Cost Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	10,067		10,067
Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Payments				86,257
Liabilities, Issuances				(233,092)
Liabilities, Begining Balance	(911,256)	(549,668)	(754,921)	(594,396)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	0	0	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	2,005	(1,481)	3,248	(1,457)
Liabilities Purchases/Payments	230,853	41,553	323,783
Liabilities, Sales/Issuance	324,639	233,092	(575,147)
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(1,003,037)	(742,688)	(1,003,037)	(742,688)
Change in unrealized gains (losses), liabilities	2,000	(1,500)	3,200	(1,500)
Financial Derivatives - Liabilities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gains (losses), liabilities	300	800	400	(100)
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	(338)	(839)	(484)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0
Liabilities, Realized Gain/(Loss)	(860)	(536)	(1,360)
Liabilities, Change In Net Unrealized Gain/(Loss)	282	762	428
Liabilities Purchases/Payments	860	494	1,360
Liabilities, Sales/Issuance	0	(42)	0
Liabilities, Transfers into Level 3	0	0	0
Liabilities, Transfers out of Level 3	0	0	0
Liabilities, Ending Balance	(56)	(77)	(56)	(77)
Securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	10,400	9,300	11,900	(87,800)
Securities | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	17,402	20,981	11,663	19,904
Assets, Accreted Discounts/Amortized Premiums	(833)	(1,525)	(1,779)	(3,245)
Assets, Realized Gain (Loss)	1,510	(1)	959	(2)
Assets, Change In Net Unrealized Gain/(Loss)	(1,389)	(592)	(1,571)	2,163
Assets, Purchases	0	3,448	555	5,660
Assets, Sales	4,528	0	(1,533)	0
Assets, Transfers into Level 3	1,897	393	4,567	411
Assets, Transfers out of Level 3	4,123	9,797	(2,925)	(11,984)
Assets, Ending Balance	9,936	12,907	9,936	12,907
Securities | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	127,329	94,197	127,838	89,581
Assets, Accreted Discounts/Amortized Premiums	604	413	1,248	510
Assets, Realized Gain (Loss)	2,576	444	2,351	406
Assets, Change In Net Unrealized Gain/(Loss)	(137)	3,282	(740)	(7,809)
Assets, Purchases	1,782	26,811	22,392	62,724
Assets, Sales	14,497	10,744	(33,076)	(27,817)
Assets, Transfers into Level 3	4,299	20,959	5,834	15,281
Assets, Transfers out of Level 3	7,492	6,755	(11,383)	(4,269)
Assets, Ending Balance	114,464	128,607	114,464	128,607
Securities | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	17,294	20,276	63,148	29,805
Assets, Accreted Discounts/Amortized Premiums	(13)	187	319	433
Assets, Realized Gain (Loss)	0	(2,253)	1,931	(867)
Assets, Change In Net Unrealized Gain/(Loss)	1,249	3,199	5,029	(8,279)
Assets, Purchases	0	2,966	0	40,235
Assets, Sales	0	3,750	(41,088)	(32,289)
Assets, Transfers into Level 3	3,353	47,397	0	44,886
Assets, Transfers out of Level 3	4,858	2,327	12,314	8,229
Assets, Ending Balance	17,025	65,695	17,025	65,695
Securities | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	37,585	43,804	111,100	44,979
Assets, Accreted Discounts/Amortized Premiums	272	(236)	1,240	(41)
Assets, Realized Gain (Loss)	(2,497)	(12,850)	(1,754)	(7,518)
Assets, Change In Net Unrealized Gain/(Loss)	9,155	4,806	16,208	(28,099)
Assets, Purchases	140	73	104	61,933
Assets, Sales	0	0	(70,002)	(1,769)
Assets, Transfers into Level 3	4,257	100,869	1,604	69,513
Assets, Transfers out of Level 3	14,923	0	(24,511)	(2,532)
Assets, Ending Balance	33,989	136,466	33,989	136,466
Securities | Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	59,473	54,627	44,925	48,610
Assets, Accreted Discounts/Amortized Premiums	(1,326)	(1,104)	(1,983)	(2,148)
Assets, Realized Gain (Loss)	146	(46)	179	(196)
Assets, Change In Net Unrealized Gain/(Loss)	(1,098)	(306)	(1,583)	(2,666)
Assets, Purchases	21,478	1,274	45,812	17,545
Assets, Sales	9,419	6,504	(18,096)	(13,204)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	69,254	47,941	69,254	47,941
Securities | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	4,761	610	4,082	1,113
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	47	208	227	208
Assets, Change In Net Unrealized Gain/(Loss)	177	578	183	483
Assets, Purchases	591	3,411	1,617	3,421
Assets, Sales	132	2,856	(665)	(3,274)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	5,444	1,951	5,444	1,951
Securities | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	4,120	712	1,590	1,394
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	0	7	(385)	7
Assets, Change In Net Unrealized Gain/(Loss)	1,711	319	2,315	(668)
Assets, Purchases	2,666	58	4,977	363
Assets, Sales	0	12	0	(12)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	8,497	1,084	8,497	1,084
Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	100	16,300	2,300	(14,100)
Loans | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,280,637	939,372	1,187,069	932,203
Assets, Accreted Discounts/Amortized Premiums	(3,240)	(1,948)	(5,711)	(2,406)
Assets, Realized Gain (Loss)	267	(781)	462	(576)
Assets, Change In Net Unrealized Gain/(Loss)	(528)	15,660	1,686	(8,663)
Assets, Purchases	327,697	70,163	549,778	201,233
Assets, Sales	159,182	74,019	(287,633)	(173,344)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	1,445,651	948,447	1,445,651	948,447
Loans | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	235,948	303,300	213,031	274,759
Assets, Accreted Discounts/Amortized Premiums	15	97	24	96
Assets, Realized Gain (Loss)	(130)	(761)	306	99
Assets, Change In Net Unrealized Gain/(Loss)	334	195	70	(134)
Assets, Purchases	90,332	1,159	147,499	88,727
Assets, Sales	98,049	8,494	(132,480)	(68,051)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	228,450	295,496	228,450	295,496
Loans | Consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	52,705	194,803	47,525	186,954
Assets, Accreted Discounts/Amortized Premiums	(1,993)	(7,901)	(3,836)	(15,371)
Assets, Realized Gain (Loss)	(11)	(11)	(1,276)	15
Assets, Change In Net Unrealized Gain/(Loss)	129	413	457	(5,338)
Assets, Purchases	9,319	15,218	26,068	76,318
Assets, Sales	8,133	35,841	(16,922)	(75,897)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	52,016	166,681	52,016	166,681
Loans | Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	13,226	6,114	5,855	18,510
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	0	0	0	0
Assets, Change In Net Unrealized Gain/(Loss)	0	0	0	0
Assets, Purchases	3,360	113	10,731	217
Assets, Sales	2	0	(2)	(12,500)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	16,584	6,227	16,584	6,227
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	147,684	65,397	141,620	71,850
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	33	0	161	0
Assets, Change In Net Unrealized Gain/(Loss)	18,569	5,643	25,076	(854)
Assets, Purchases	69,464	1,933	82,333	14,216
Assets, Sales	56,771	420	(70,211)	(12,659)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	178,979	72,553	178,979	72,553
Change in unrealized gain (loss), assets	16,400	5,500	22,700	(1,100)
Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	400	400	500	900
Financial Derivatives - Assets | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	2	37	9	620
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	0	51	141	242
Assets, Change In Net Unrealized Gain/(Loss)	518	405	512	(178)
Assets, Purchases	0	0	0	0
Assets, Sales	0	51	(142)	(242)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	520	442	520	442
Financial Derivatives - Assets | Credit default swaps on asset-backed securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	320	353	347	993
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	16	38	42	(956)
Assets, Change In Net Unrealized Gain/(Loss)	(17)	0	(44)	916
Assets, Purchases	3	21	7	26
Assets, Sales	19	59	(49)	(626)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	$ 303	353	$ 303	353
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Payments				504
Liabilities, Issuances				0
Liabilities, Begining Balance				(436)
Liabilities, Accreted Discounts/Amortized Premiums				0
Liabilities, Realized Gain/(Loss)				(504)
Liabilities, Change In Net Unrealized Gain/(Loss)				359
Liabilities, Transfers into Level 3				0
Liabilities, Transfers out of Level 3				0
Liabilities, Ending Balance		$ (77)		$ (77)