Valuation (Significant Unobservable Inputs Rollforward) (Details) - Level 3 - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 1,721,275	$ 1,094,815
Assets, Accreted Discounts/Amortized Premiums	(10,680)	(10,641)
Assets, Realized Gain (Loss)	1,366	(1,647)
Assets, Change In Net Unrealized Gain/(Loss)	(80,188)	3,447
Assets, Purchases	401,709	303,325
Assets, Sales	(274,433)	(167,106)
Assets, Transfers into Level 3	15,143	15,690
Assets, Transfers out of Level 3	(29,609)	(4,564)
Assets, Ending Balance	1,744,583	1,233,319
Liabilities, Begining Balance	(594,832)	(297,948)
Liabilities, Accreted Discounts/Amortized Premiums	0	0
Liabilities, Realized Gain/(Loss)	31	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(379)	57
Liabilities Purchases/Payments	44,714	15,767
Liabilities, Sales/Issuance	(41)	0
Liabilities, Transfers into Level 3	0	0
Liabilities, Transfers out of Level 3	0	0
Liabilities, Ending Balance	(550,507)	(282,124)
Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	(594,396)	(297,948)
Liabilities, Accreted Discounts/Amortized Premiums	0	0
Liabilities, Realized Gain/(Loss)	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	24	57
Liabilities, Payments	44,704	15,767
Liabilities, Issuances	0	0
Liabilities, Transfers into Level 3	0	0
Liabilities, Transfers out of Level 3	0	0
Liabilities, Ending Balance	(549,668)	(282,124)
Change in unrealized gains (losses), liabilities	24	57
Financial Derivatives - Liabilities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gains (losses), liabilities	(800)
Securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	(50,900)	700
Securities | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	19,904	7,293
Assets, Accreted Discounts/Amortized Premiums	(1,822)	(774)
Assets, Realized Gain (Loss)	(1)	(594)
Assets, Change In Net Unrealized Gain/(Loss)	2,807	189
Assets, Purchases	5,259	6
Assets, Sales	0	0
Assets, Transfers into Level 3	1,088	842
Assets, Transfers out of Level 3	(6,254)	(573)
Assets, Ending Balance	20,981	6,389
Securities | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	89,581	91,291
Assets, Accreted Discounts/Amortized Premiums	226	63
Assets, Realized Gain (Loss)	(136)	(101)
Assets, Change In Net Unrealized Gain/(Loss)	(11,533)	(535)
Assets, Purchases	33,950	15,546
Assets, Sales	(14,395)	(19,436)
Assets, Transfers into Level 3	3,659	10,492
Assets, Transfers out of Level 3	(7,155)	(2,650)
Assets, Ending Balance	94,197	94,670
Securities | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	29,805	803
Assets, Accreted Discounts/Amortized Premiums	207	(14)
Assets, Realized Gain (Loss)	1,386	0
Assets, Change In Net Unrealized Gain/(Loss)	(11,193)	(8)
Assets, Purchases	31,025	0
Assets, Sales	(28,539)	0
Assets, Transfers into Level 3	4,071	4,356
Assets, Transfers out of Level 3	6,486	0
Assets, Ending Balance	20,276	5,137
Securities | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	44,979	14,915
Assets, Accreted Discounts/Amortized Premiums	(318)	(406)
Assets, Realized Gain (Loss)	(21)	(83)
Assets, Change In Net Unrealized Gain/(Loss)	(20,261)	49
Assets, Purchases	22,760	8,304
Assets, Sales	54	0
Assets, Transfers into Level 3	6,325	0
Assets, Transfers out of Level 3	(9,714)	(1,341)
Assets, Ending Balance	43,804	21,438
Securities | Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	48,610	22,800
Assets, Accreted Discounts/Amortized Premiums	(1,044)	(609)
Assets, Realized Gain (Loss)	(150)	(512)
Assets, Change In Net Unrealized Gain/(Loss)	(2,360)	762
Assets, Purchases	16,271	4,940
Assets, Sales	(6,700)	(3,273)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	54,627	24,108
Securities | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,113	6,318
Assets, Accreted Discounts/Amortized Premiums	0	16
Assets, Realized Gain (Loss)	0	(1)
Assets, Change In Net Unrealized Gain/(Loss)	(96)	(77)
Assets, Purchases	10	384
Assets, Sales	(417)	(903)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	610	5,737
Securities | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,394	1,530
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	0	0
Assets, Change In Net Unrealized Gain/(Loss)	(987)	(65)
Assets, Purchases	305	0
Assets, Sales	0	0
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	712	1,465
Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	(30,400)	3,400
Loans | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	932,203	496,829
Assets, Accreted Discounts/Amortized Premiums	(458)	(927)
Assets, Realized Gain (Loss)	205	(136)
Assets, Change In Net Unrealized Gain/(Loss)	(24,323)	1,901
Assets, Purchases	131,070	157,602
Assets, Sales	(99,325)	(72,017)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	939,372	583,252
Loans | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	274,759	195,301
Assets, Accreted Discounts/Amortized Premiums	(1)	306
Assets, Realized Gain (Loss)	860	0
Assets, Change In Net Unrealized Gain/(Loss)	(328)	(333)
Assets, Purchases	87,567	48,857
Assets, Sales	(59,557)	(4,508)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	303,300	239,623
Loans | Consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	186,954	183,961
Assets, Accreted Discounts/Amortized Premiums	(7,470)	(8,572)
Assets, Realized Gain (Loss)	26	(2,055)
Assets, Change In Net Unrealized Gain/(Loss)	(5,751)	1,842
Assets, Purchases	61,100	54,256
Assets, Sales	(40,056)	(37,317)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	194,803	192,115
Loans | Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	18,510	72,302
Assets, Accreted Discounts/Amortized Premiums	0	276
Assets, Realized Gain (Loss)	0	1,560
Assets, Change In Net Unrealized Gain/(Loss)	0	(39)
Assets, Purchases	104	13,428
Assets, Sales	(12,500)	(29,375)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	6,114	58,152
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	71,850
Assets, Accreted Discounts/Amortized Premiums	0
Assets, Realized Gain (Loss)	0
Assets, Change In Net Unrealized Gain/(Loss)	(6,497)
Assets, Purchases	12,283
Assets, Sales	(12,239)
Assets, Transfers into Level 3	0
Assets, Transfers out of Level 3	0
Assets, Ending Balance	65,397
Change in unrealized gain (loss), assets	(6,700)	(2,100)
Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	500	(200)
Financial Derivatives - Assets | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	620
Assets, Accreted Discounts/Amortized Premiums	0
Assets, Realized Gain (Loss)	191
Assets, Change In Net Unrealized Gain/(Loss)	(583)
Assets, Purchases	0
Assets, Sales	(191)
Assets, Transfers into Level 3	0
Assets, Transfers out of Level 3	0
Assets, Ending Balance	37
Financial Derivatives - Assets | Credit default swaps on asset-backed securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	993	1,472
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	(994)	275
Assets, Change In Net Unrealized Gain/(Loss)	917	(239)
Assets, Purchases	5	2
Assets, Sales	(568)	(277)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	353	$ 1,233
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	(436)
Liabilities, Accreted Discounts/Amortized Premiums	0
Liabilities, Realized Gain/(Loss)	31
Liabilities, Change In Net Unrealized Gain/(Loss)	(403)
Liabilities, Payments	10
Liabilities, Issuances	(41)
Liabilities, Transfers into Level 3	0
Liabilities, Transfers out of Level 3	0
Liabilities, Ending Balance	$ (839)