Valuation (Significant Unobservable Inputs Rollforward) (Details) - Level 3 - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2021	Mar. 31, 2020
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 1,959,802	$ 1,721,275
Assets, Accreted Discounts/Amortized Premiums	(4,451)	(10,680)
Assets, Realized Gain (Loss)	2,230	1,366
Assets, Change In Net Unrealized Gain/(Loss)	16,042	(80,188)
Assets, Purchases	368,281	401,709
Assets, Sales	(296,047)	(274,433)
Assets, Transfers into Level 3	9,366	15,143
Assets, Transfers out of Level 3	(56,737)	(29,609)
Assets, Ending Balance	1,998,486	1,744,583
Liabilities, Begining Balance	(755,405)	(594,832)
Liabilities, Accreted Discounts/Amortized Premiums	0	0
Liabilities, Realized Gain/(Loss)	(500)	31
Liabilities, Change In Net Unrealized Gain/(Loss)	1,389	(379)
Liabilities Purchases/Payments	93,430	44,714
Liabilities, Sales/Issuance	(250,508)	(41)
Liabilities, Transfers into Level 3	0	0
Liabilities, Transfers out of Level 3	0	0
Liabilities, Ending Balance	(911,594)	(550,507)
Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Payments		44,704
Liabilities, Issuances		0
Liabilities, Begining Balance	(754,921)	(594,396)
Liabilities, Accreted Discounts/Amortized Premiums	0	0
Liabilities, Realized Gain/(Loss)	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	1,243	24
Liabilities Purchases/Payments	92,930
Liabilities, Sales/Issuance	(250,508)
Liabilities, Transfers into Level 3	0	0
Liabilities, Transfers out of Level 3	0	0
Liabilities, Ending Balance	(911,256)	(549,668)
Change in unrealized gains (losses), liabilities	1,200	24
Financial Derivatives - Liabilities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gains (losses), liabilities	100	(800)
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	(484)
Liabilities, Accreted Discounts/Amortized Premiums	0
Liabilities, Realized Gain/(Loss)	(500)
Liabilities, Change In Net Unrealized Gain/(Loss)	146
Liabilities Purchases/Payments	500
Liabilities, Sales/Issuance	0
Liabilities, Transfers into Level 3	0
Liabilities, Transfers out of Level 3	0
Liabilities, Ending Balance	(338)
Securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	3,200	(50,900)
Securities | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	11,663	19,904
Assets, Accreted Discounts/Amortized Premiums	(1,121)	(1,822)
Assets, Realized Gain (Loss)	(26)	(1)
Assets, Change In Net Unrealized Gain/(Loss)	(136)	2,807
Assets, Purchases	1,814	5,259
Assets, Sales	0	0
Assets, Transfers into Level 3	5,857	1,088
Assets, Transfers out of Level 3	(649)	(6,254)
Assets, Ending Balance	17,402	20,981
Securities | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	127,838	89,581
Assets, Accreted Discounts/Amortized Premiums	679	226
Assets, Realized Gain (Loss)	(226)	(136)
Assets, Change In Net Unrealized Gain/(Loss)	(557)	(11,533)
Assets, Purchases	20,742	33,950
Assets, Sales	(18,639)	(14,395)
Assets, Transfers into Level 3	1,998	3,659
Assets, Transfers out of Level 3	(4,506)	(7,155)
Assets, Ending Balance	127,329	94,197
Securities | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	63,148	29,805
Assets, Accreted Discounts/Amortized Premiums	218	207
Assets, Realized Gain (Loss)	2,082	1,386
Assets, Change In Net Unrealized Gain/(Loss)	2,525	(11,193)
Assets, Purchases	0	31,025
Assets, Sales	(39,705)	(28,539)
Assets, Transfers into Level 3	0	4,071
Assets, Transfers out of Level 3	10,974	6,486
Assets, Ending Balance	17,294	20,276
Securities | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	111,100	44,979
Assets, Accreted Discounts/Amortized Premiums	734	(318)
Assets, Realized Gain (Loss)	912	(21)
Assets, Change In Net Unrealized Gain/(Loss)	5,334	(20,261)
Assets, Purchases	1,812	22,760
Assets, Sales	(43,210)	54
Assets, Transfers into Level 3	1,511	6,325
Assets, Transfers out of Level 3	(40,608)	(9,714)
Assets, Ending Balance	37,585	43,804
Securities | Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	44,925	48,610
Assets, Accreted Discounts/Amortized Premiums	(657)	(1,044)
Assets, Realized Gain (Loss)	33	(150)
Assets, Change In Net Unrealized Gain/(Loss)	(485)	(2,360)
Assets, Purchases	24,334	16,271
Assets, Sales	(8,677)	(6,700)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	59,473	54,627
Securities | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	4,082	1,113
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	180	0
Assets, Change In Net Unrealized Gain/(Loss)	5	(96)
Assets, Purchases	1,027	10
Assets, Sales	(533)	(417)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	4,761	610
Securities | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,590	1,394
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	(385)	0
Assets, Change In Net Unrealized Gain/(Loss)	604	(987)
Assets, Purchases	2,311	305
Assets, Sales	0	0
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	4,120	712
Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	2,000	(30,400)
Loans | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,187,069	932,203
Assets, Accreted Discounts/Amortized Premiums	(2,470)	(458)
Assets, Realized Gain (Loss)	194	205
Assets, Change In Net Unrealized Gain/(Loss)	2,214	(24,323)
Assets, Purchases	222,081	131,070
Assets, Sales	(128,451)	(99,325)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	1,280,637	939,372
Loans | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	213,031	274,759
Assets, Accreted Discounts/Amortized Premiums	9	(1)
Assets, Realized Gain (Loss)	436	860
Assets, Change In Net Unrealized Gain/(Loss)	(263)	(328)
Assets, Purchases	57,166	87,567
Assets, Sales	(34,431)	(59,557)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	235,948	303,300
Loans | Consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	47,525	186,954
Assets, Accreted Discounts/Amortized Premiums	(1,843)	(7,470)
Assets, Realized Gain (Loss)	(1,265)	26
Assets, Change In Net Unrealized Gain/(Loss)	327	(5,751)
Assets, Purchases	16,749	61,100
Assets, Sales	(8,788)	(40,056)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	52,705	194,803
Loans | Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	5,855	18,510
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	0	0
Assets, Change In Net Unrealized Gain/(Loss)	0	0
Assets, Purchases	7,371	104
Assets, Sales	0	(12,500)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	13,226	6,114
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	141,620	71,850
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	128	0
Assets, Change In Net Unrealized Gain/(Loss)	6,507	(6,497)
Assets, Purchases	12,870	12,283
Assets, Sales	(13,441)	(12,239)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	147,684	65,397
Change in unrealized gain (loss), assets	6,300	(6,700)
Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	(33)	500
Financial Derivatives - Assets | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	9	620
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	141	191
Assets, Change In Net Unrealized Gain/(Loss)	(6)	(583)
Assets, Purchases	0	0
Assets, Sales	(142)	(191)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	2	37
Financial Derivatives - Assets | Credit default swaps on asset-backed securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	347	993
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	26	(994)
Assets, Change In Net Unrealized Gain/(Loss)	(27)	917
Assets, Purchases	4	5
Assets, Sales	(30)	(568)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	$ 320	353
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Payments		10
Liabilities, Issuances		(41)
Liabilities, Begining Balance		(436)
Liabilities, Accreted Discounts/Amortized Premiums		0
Liabilities, Realized Gain/(Loss)		31
Liabilities, Change In Net Unrealized Gain/(Loss)		(403)
Liabilities, Transfers into Level 3		0
Liabilities, Transfers out of Level 3		0
Liabilities, Ending Balance		$ (839)