Valuation (Significant Unobservable Inputs Rollforward) (Details) - Level 3 - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2020	Sep. 30, 2019	Sep. 30, 2020	Sep. 30, 2019
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 1,884,850	$ 1,324,591	$ 1,721,275	$ 1,094,815
Assets, Accreted Discounts/Amortized Premiums	(7,746)	(10,822)	(32,702)	(29,952)
Assets, Realized Gain (Loss)	347	5,702	(8,689)	3,262
Assets, Change In Net Unrealized Gain/(Loss)	35,608	1,047	(18,457)	13,268
Assets, Purchases	250,453	330,413	782,266	814,294
Assets, Sales	188,359	173,334	(604,788)	(433,042)
Assets, Transfers into Level 3	20,559	22,484	86,441	38,490
Assets, Transfers out of Level 3	99,184	12,357	(28,818)	(13,411)
Assets, Ending Balance	1,896,528	1,487,724	1,896,528	1,487,724
Liabilities, Begining Balance	742,765	(475,816)	(594,832)	(297,948)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	4	48	(501)	48
Liabilities, Change In Net Unrealized Gain/(Loss)	(5,189)	(286)	(6,287)	(246)
Liabilities Purchases/Payments	52,346	37,259	139,108	78,887
Liabilities, Sales/Issuance	0	(48)	(233,092)	(219,584)
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(695,604)	(438,843)	(695,604)	(438,843)
Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	742,688	(475,816)	(594,396)	(297,948)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	0	0	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(5,178)	(72)	(6,636)	(32)
Liabilities, Payments	52,350	37,259	138,608	78,887
Liabilities, Issuances	0	0	(233,092)	(219,536)
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(695,516)	(438,629)	(695,516)	(438,629)
Change in unrealized gains (losses), liabilities	(5,200)	(100)	(6,600)	(32)
Financial Derivatives - Liabilities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gains (losses), liabilities	(11)	(200)	(100)	(200)
Securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	9,300	(4,700)	(56,300)	(400)
Securities | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	12,907	11,034	19,904	7,293
Assets, Accreted Discounts/Amortized Premiums	(1,482)	(1,164)	(6,072)	(2,228)
Assets, Realized Gain (Loss)	494	(621)	614	(1,228)
Assets, Change In Net Unrealized Gain/(Loss)	280	696	2,803	1,430
Assets, Purchases	3,490	13,254	8,219	14,854
Assets, Sales	2,243	0	(3,295)	(463)
Assets, Transfers into Level 3	2,621	737	2,177	4,708
Assets, Transfers out of Level 3	3,927	437	(12,210)	(867)
Assets, Ending Balance	12,140	23,499	12,140	23,499
Securities | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	128,607	96,790	89,581	91,291
Assets, Accreted Discounts/Amortized Premiums	884	(121)	1,113	104
Assets, Realized Gain (Loss)	(77)	7,836	303	7,187
Assets, Change In Net Unrealized Gain/(Loss)	2,610	(6,457)	(1,996)	(4,482)
Assets, Purchases	21,138	6,483	61,933	10,239
Assets, Sales	6,395	21,457	(29,811)	(27,514)
Assets, Transfers into Level 3	10,895	5,808	22,974	14,633
Assets, Transfers out of Level 3	18,463	7,400	(4,898)	(9,976)
Assets, Ending Balance	139,199	81,482	139,199	81,482
Securities | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	65,695	6,278	29,805	803
Assets, Accreted Discounts/Amortized Premiums	659	25	617	(16)
Assets, Realized Gain (Loss)	8	374	(867)	76
Assets, Change In Net Unrealized Gain/(Loss)	4,117	(302)	(4,638)	(73)
Assets, Purchases	12,699	6,815	51,432	7,937
Assets, Sales	8	1,474	(32,284)	(221)
Assets, Transfers into Level 3	560	652	7,828	3,862
Assets, Transfers out of Level 3	40,921	0	9,084	0
Assets, Ending Balance	42,809	12,368	42,809	12,368
Securities | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	136,466	17,222	44,979	14,915
Assets, Accreted Discounts/Amortized Premiums	1,962	(184)	1,237	(670)
Assets, Realized Gain (Loss)	(82)	1,158	(7,586)	(536)
Assets, Change In Net Unrealized Gain/(Loss)	1,920	184	(24,206)	1,889
Assets, Purchases	2,241	0	47,730	816
Assets, Sales	127	1,139	0	(1,125)
Assets, Transfers into Level 3	6,483	15,287	53,462	15,287
Assets, Transfers out of Level 3	35,873	4,520	(2,626)	(2,568)
Assets, Ending Balance	112,990	28,008	112,990	28,008
Securities | Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	47,941	25,019	48,610	22,800
Assets, Accreted Discounts/Amortized Premiums	(1,435)	(611)	(3,584)	(1,580)
Assets, Realized Gain (Loss)	(4)	(100)	(200)	(765)
Assets, Change In Net Unrealized Gain/(Loss)	1,165	(320)	(1,501)	537
Assets, Purchases	7,455	18,638	25,001	28,189
Assets, Sales	7,824	3,310	(21,028)	(9,865)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	47,298	39,316	47,298	39,316
Securities | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,951	4,081	1,113	6,318
Assets, Accreted Discounts/Amortized Premiums	0	0	0	22
Assets, Realized Gain (Loss)	296	(583)	504	(928)
Assets, Change In Net Unrealized Gain/(Loss)	509	142	991	65
Assets, Purchases	28	6,425	3,450	9,257
Assets, Sales	577	6,883	(3,851)	(11,552)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	2,207	3,182	2,207	3,182
Securities | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,084	1,791	1,394	1,534
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	0	(768)	7	(910)
Assets, Change In Net Unrealized Gain/(Loss)	(73)	(61)	(741)	337
Assets, Purchases	3	1,316	366	1,317
Assets, Sales	0	0	(12)	0
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	1,014	2,278	1,014	2,278
Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	14,200	4,100	12	10,200
Loans | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	948,447	663,880	932,203	496,829
Assets, Accreted Discounts/Amortized Premiums	(2,009)	(2,241)	(4,414)	(4,515)
Assets, Realized Gain (Loss)	(146)	(400)	(722)	1,554
Assets, Change In Net Unrealized Gain/(Loss)	15,672	3,559	7,008	7,717
Assets, Purchases	139,609	191,512	340,842	452,958
Assets, Sales	70,864	58,582	(244,208)	(156,815)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	1,030,709	797,728	1,030,709	797,728
Loans | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	295,496	260,034	274,759	195,301
Assets, Accreted Discounts/Amortized Premiums	17	(52)	113	1,087
Assets, Realized Gain (Loss)	36	(1)	135	1,412
Assets, Change In Net Unrealized Gain/(Loss)	(6)	507	(140)	(1,844)
Assets, Purchases	18,881	32,426	107,608	128,839
Assets, Sales	62,193	32,288	(130,244)	(64,169)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	252,231	260,626	252,231	260,626
Loans | Consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	166,681	162,609	186,954	183,961
Assets, Accreted Discounts/Amortized Premiums	(6,342)	(6,474)	(21,712)	(22,432)
Assets, Realized Gain (Loss)	(112)	(1,055)	(97)	(4,565)
Assets, Change In Net Unrealized Gain/(Loss)	(1,627)	28	(6,965)	2,726
Assets, Purchases	28,554	33,101	104,872	103,983
Assets, Sales	34,545	36,510	(110,443)	(111,974)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	152,609	151,699	152,609	151,699
Loans | Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	6,227	5,000	18,510	0
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	0	0	0	0
Assets, Change In Net Unrealized Gain/(Loss)	0	0	0	0
Assets, Purchases	836	10,790	1,053	15,790
Assets, Sales	0	0	(12,500)	0
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	7,063	15,790	7,063	15,790
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	72,553	69,676	71,850	72,298
Assets, Accreted Discounts/Amortized Premiums	0	0	0	276
Assets, Realized Gain (Loss)	62	(139)	62	1,545
Assets, Change In Net Unrealized Gain/(Loss)	11,381	2,935	10,528	5,125
Assets, Purchases	15,307	9,643	29,522	40,097
Assets, Sales	3,500	11,680	(16,159)	(48,906)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	95,803	70,435	95,803	70,435
Change in unrealized gain (loss), assets	11,300	3,300	10,200	2,400
Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	(300)	100	600	(200)
Financial Derivatives - Assets | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	442	87	620	0
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	(129)	(15)	113	1
Assets, Change In Net Unrealized Gain/(Loss)	(338)	148	(515)	235
Assets, Purchases	209	0	209	0
Assets, Sales	79	(15)	(322)	(1)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	105	235	105	235
Financial Derivatives - Assets | Credit default swaps on asset-backed securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	353	1,090	993	1,472
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	1	16	(955)	419
Assets, Change In Net Unrealized Gain/(Loss)	(2)	(12)	915	(394)
Assets, Purchases	3	10	29	18
Assets, Sales	4	26	(631)	(437)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	351	1,078	351	1,078
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	77	0	(436)	0
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	4	48	(501)	48
Liabilities, Change In Net Unrealized Gain/(Loss)	(11)	(214)	349	(214)
Liabilities, Payments	(4)	0	500	0
Liabilities, Issuances	0	(48)	0	(48)
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	$ (88)	$ (214)	$ (88)	$ (214)