Valuation (Significant Unobservable Inputs Rollforward) (Details) - Level 3 - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2021	Sep. 30, 2020	Sep. 30, 2021	Sep. 30, 2020	Dec. 31, 2020
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 2,181,112	$ 1,884,850	$ 1,959,802	$ 1,721,275
Assets, Accreted Discounts/Amortized Premiums	(9,930)	(7,746)	(19,714)	(32,702)
Assets, Realized Gain (Loss)	11,996	347	15,710	(8,689)
Assets, Change In Net Unrealized Gain/(Loss)	(8,023)	35,608	40,536	(18,457)
Assets, Purchases	522,267	250,453	1,413,347	782,266
Assets, Sales	325,302	188,359	(1,001,333)	(604,788)
Assets, Transfers into Level 3	11,040	20,559	12,922	86,441
Assets, Transfers out of Level 3	0	99,184	(38,110)	(28,818)
Assets, Ending Balance	2,383,160	1,896,528	2,383,160	1,896,528
Liabilities, Begining Balance	(1,003,093)	(742,765)	(755,405)	(594,832)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	(67)	4	(1,428)	(501)
Liabilities, Change In Net Unrealized Gain/(Loss)	1,538	(5,189)	5,214	(6,287)
Liabilities Purchases/Payments	129,316	52,346	454,460	139,108
Liabilities, Sales/Issuance	0	0	(575,147)	(233,092)
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(872,306)	(695,604)	(872,306)	(695,604)
Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	106,600		106,600		$ 127,838
Non-Agency RMBS | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	37,510		37,510		57,219
CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	12,148		12,148		63,148
CMBS | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	269		269		9,949
CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	30,121		30,121		111,100
CLOs | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	5,922		5,922		8,190
Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets					44,925
Asset-backed securities, backed by consumer loans | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	71,295		71,295		44,828
Consumer loans | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	55,676		55,676		47,525
Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	16,561		16,561
Corporate loan [Member] | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	9,561		9,561
Total return swaps | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets					9
Credit default swaps on asset-backed securities | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	303		303		347
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	142,019		142,019		141,620
Investment in unconsolidated entities | Measurement Input, Net Asset Value | Enterprise Value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	38,613		38,613		$ 62,084
Investment in unconsolidated entities | Measurement Input, Exercise Price | Valuation, Cost Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	10,568		10,568
Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Payments				138,608
Liabilities, Issuances				(233,092)
Liabilities, Begining Balance	(1,003,037)	(742,688)	(754,921)	(594,396)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	0	0	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	1,482	(5,178)	4,730	(6,636)
Liabilities Purchases/Payments	129,249	52,350	453,032
Liabilities, Sales/Issuance	0	0	(575,147)
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(872,306)	(695,516)	(872,306)	(695,516)
Change in unrealized gains (losses), liabilities	1,500	(5,200)	4,700	(6,600)
Financial Derivatives - Liabilities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gains (losses), liabilities	100	(11)		(100)
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	(56)	(77)	(484)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0
Liabilities, Realized Gain/(Loss)	(67)	4	(1,428)
Liabilities, Change In Net Unrealized Gain/(Loss)	56	(11)	484
Liabilities Purchases/Payments	67	(4)	1,428
Liabilities, Sales/Issuance	0	0	0
Liabilities, Transfers into Level 3	0	0	0
Liabilities, Transfers out of Level 3	0	0	0
Liabilities, Ending Balance	0	(88)	0	(88)
Securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	3,100	9,300	13,600	(56,300)
Securities | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	9,936	12,907	11,663	19,904
Assets, Accreted Discounts/Amortized Premiums	(1,285)	(1,482)	(2,570)	(6,072)
Assets, Realized Gain (Loss)	(15)	494	943	614
Assets, Change In Net Unrealized Gain/(Loss)	278	280	(1,408)	2,803
Assets, Purchases	2,038	3,490	2,039	8,219
Assets, Sales	0	2,243	(1,533)	(3,295)
Assets, Transfers into Level 3	780	2,621	5,093	2,177
Assets, Transfers out of Level 3	0	3,927	(2,495)	(12,210)
Assets, Ending Balance	11,732	12,140	11,732	12,140
Securities | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	114,464	128,607	127,838	89,581
Assets, Accreted Discounts/Amortized Premiums	612	884	1,926	1,113
Assets, Realized Gain (Loss)	680	(77)	2,993	303
Assets, Change In Net Unrealized Gain/(Loss)	(1,918)	2,610	(2,533)	(1,996)
Assets, Purchases	154	21,138	22,392	61,933
Assets, Sales	9,005	6,395	(42,009)	(29,811)
Assets, Transfers into Level 3	1,613	10,895	5,831	22,974
Assets, Transfers out of Level 3	0	18,463	(9,838)	(4,898)
Assets, Ending Balance	106,600	139,199	106,600	139,199
Securities | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	17,025	65,695	63,148	29,805
Assets, Accreted Discounts/Amortized Premiums	27	659	404	617
Assets, Realized Gain (Loss)	1,987	8	4,329	(867)
Assets, Change In Net Unrealized Gain/(Loss)	(468)	4,117	4,946	(4,638)
Assets, Purchases	0	12,699	0	51,432
Assets, Sales	15,070	8	(60,449)	(32,284)
Assets, Transfers into Level 3	8,647	560	397	7,828
Assets, Transfers out of Level 3	0	40,921	627	9,084
Assets, Ending Balance	12,148	42,809	12,148	42,809
Securities | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	33,989	136,466	111,100	44,979
Assets, Accreted Discounts/Amortized Premiums	(1,688)	1,962	(371)	1,237
Assets, Realized Gain (Loss)	2,689	(82)	934	(7,586)
Assets, Change In Net Unrealized Gain/(Loss)	3,007	1,920	19,782	(24,206)
Assets, Purchases	85	2,241	104	47,730
Assets, Sales	7,961	127	(77,879)	0
Assets, Transfers into Level 3	0	6,483	1,601	53,462
Assets, Transfers out of Level 3	0	35,873	(25,150)	(2,626)
Assets, Ending Balance	30,121	112,990	30,121	112,990
Securities | Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	69,254	47,941	44,925	48,610
Assets, Accreted Discounts/Amortized Premiums	(1,813)	(1,435)	(3,796)	(3,584)
Assets, Realized Gain (Loss)	287	(4)	466	(200)
Assets, Change In Net Unrealized Gain/(Loss)	(1,598)	1,165	(3,181)	(1,501)
Assets, Purchases	15,853	7,455	61,665	25,001
Assets, Sales	10,688	7,824	(28,784)	(21,028)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	71,295	47,298	71,295	47,298
Securities | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	5,444	1,951	4,082	1,113
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	1,271	296	1,497	504
Assets, Change In Net Unrealized Gain/(Loss)	(700)	509	(517)	991
Assets, Purchases	2,207	28	3,824	3,450
Assets, Sales	3,803	577	(4,467)	(3,851)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	4,419	2,207	4,419	2,207
Securities | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	8,497	1,084	1,590	1,394
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	661	0	276	7
Assets, Change In Net Unrealized Gain/(Loss)	224	(73)	2,538	(741)
Assets, Purchases	351	3	5,329	366
Assets, Sales	1,739	0	(1,739)	(12)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	7,994	1,014	7,994	1,014
Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	(4,700)	14,200	(2,400)	12
Loans | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,445,651	948,447	1,187,069	932,203
Assets, Accreted Discounts/Amortized Premiums	(3,864)	(2,009)	(9,575)	(4,414)
Assets, Realized Gain (Loss)	277	(146)	738	(722)
Assets, Change In Net Unrealized Gain/(Loss)	(4,475)	15,672	(2,788)	7,008
Assets, Purchases	405,291	139,609	955,069	340,842
Assets, Sales	186,179	70,864	(473,812)	(244,208)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	1,656,701	1,030,709	1,656,701	1,030,709
Loans | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	228,450	295,496	213,031	274,759
Assets, Accreted Discounts/Amortized Premiums	0	17	24	113
Assets, Realized Gain (Loss)	17	36	323	135
Assets, Change In Net Unrealized Gain/(Loss)	71	(6)	141	(140)
Assets, Purchases	72,415	18,881	219,914	107,608
Assets, Sales	33,362	62,193	(165,842)	(130,244)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	267,591	252,231	267,591	252,231
Loans | Consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	52,016	166,681	47,525	186,954
Assets, Accreted Discounts/Amortized Premiums	(1,919)	(6,342)	(5,756)	(21,712)
Assets, Realized Gain (Loss)	(105)	(112)	(1,381)	(97)
Assets, Change In Net Unrealized Gain/(Loss)	19	(1,627)	476	(6,965)
Assets, Purchases	13,941	28,554	40,008	104,872
Assets, Sales	8,276	34,545	(25,196)	(110,443)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	55,676	152,609	55,676	152,609
Loans | Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	16,584	6,227	5,855	18,510
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	0	0	0	0
Assets, Change In Net Unrealized Gain/(Loss)	(267)	0	(267)	0
Assets, Purchases	254	836	10,985	1,053
Assets, Sales	10	0	(12)	(12,500)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	16,561	7,063	16,561	7,063
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	178,979	72,553	141,620	71,850
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	4,225	62	4,386	62
Assets, Change In Net Unrealized Gain/(Loss)	(1,676)	11,381	23,400	10,528
Assets, Purchases	9,674	15,307	92,008	29,522
Assets, Sales	49,183	3,500	(119,395)	(16,159)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	142,019	95,803	142,019	95,803
Change in unrealized gain (loss), assets	(2,700)	11,300	19,900	10,200
Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	100	(300)	(100)	600
Financial Derivatives - Assets | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	520	442	9	620
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	26	(129)	167	113
Assets, Change In Net Unrealized Gain/(Loss)	(520)	(338)	(9)	(515)
Assets, Purchases	0	209	0	209
Assets, Sales	26	79	(167)	(322)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	0	105	0	105
Financial Derivatives - Assets | Credit default swaps on asset-backed securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	303	353	347	993
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	(4)	1	39	(955)
Assets, Change In Net Unrealized Gain/(Loss)	0	(2)	(44)	915
Assets, Purchases	4	3	10	29
Assets, Sales	0	4	(49)	(631)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	$ 303	351	$ 303	351
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Payments				500
Liabilities, Issuances				0
Liabilities, Begining Balance				(436)
Liabilities, Accreted Discounts/Amortized Premiums				0
Liabilities, Realized Gain/(Loss)				(501)
Liabilities, Change In Net Unrealized Gain/(Loss)				349
Liabilities, Transfers into Level 3				0
Liabilities, Transfers out of Level 3				0
Liabilities, Ending Balance		$ (88)		$ (88)