Valuation (Significant Unobservable Inputs Rollforward) (Details) - Level 3 - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2022	Mar. 31, 2021	Dec. 31, 2021
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 2,881,539	$ 1,959,802
Assets, Accreted Discounts/Amortized Premiums	(8,488)	(4,451)
Assets, Realized Gain (Loss)	7,433	2,230
Assets, Change In Net Unrealized Gain/(Loss)	(84,296)	16,042
Assets, Purchases	1,174,093	368,281
Assets, Sales	(604,284)	(296,047)
Assets, Transfers into Level 3	13,300	9,366
Assets, Transfers out of Level 3	(30,471)	(56,737)
Assets, Ending Balance	3,348,826	1,998,486
Liabilities, Begining Balance	(984,168)	(755,405)
Liabilities, Accreted Discounts/Amortized Premiums	0	0
Liabilities, Realized Gain/(Loss)	0	(500)
Liabilities, Change In Net Unrealized Gain/(Loss)	55,641	1,326
Liabilities Purchases/Payments	114,754	93,493
Liabilities, Sales/Issuance	(612,769)	(250,508)
Liabilities, Transfers into Level 3	0	0
Liabilities, Transfers out of Level 3	0	0
Liabilities, Ending Balance	(1,426,542)	(911,594)
Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	116,776		$ 134,888
Non-Agency RMBS | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	35,883		45,544
Non-Agency RMBS | Valuation, Market Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	80,893		89,344
CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	9,526		13,134
CMBS | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	892		268
CMBS | Valuation, Market Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	8,634		12,866
CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	22,824		26,678
CLOs | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	4,772		8,014
CLOs | Valuation, Market Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	18,052		18,664
Asset-backed securities, backed by consumer loans | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	76,504		73,108
Residential mortgage loans | Performing Financial Instruments [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	1,096,174		952,656
Residential mortgage loans | Valuation, Income Approach | Performing Financial Instruments [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	1,060,745		951,723
Residential mortgage loans | Measurement Input, Exercise Price | Valuation, Cost Approach | Performing Financial Instruments [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	35,429		933
Commercial mortgage loans | Valuation, Income Approach | Performing Financial Instruments [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	398,494		310,735
Consumer loans | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	9,878		62,365
Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	11,788		10,531
Corporate loan [Member] | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	4,788		3,531
Corporate loan [Member] | Valuation, Market Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	7,000		7,000
Credit default swaps on asset-backed securities | Valuation, Income Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	304		303
Other secured borrowings, at fair value | Valuation, Market Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Fair value	(1,216,542)		(984,168)
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	219,303		195,643
Investment in unconsolidated entities | Measurement Input, Net Asset Value | Enterprise Value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	89,033		57,828
Investment in unconsolidated entities | Measurement Input, Exercise Price | Valuation, Cost Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Total assets	14,036		$ 14,036
Senior notes | Valuation, Cost Approach
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Fair value	(210,000)
Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Payments		92,993
Liabilities, Issuances		(250,508)
Liabilities, Begining Balance	(984,168)	(754,921)
Liabilities, Accreted Discounts/Amortized Premiums	0	0
Liabilities, Realized Gain/(Loss)	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	55,641	1,180
Liabilities Purchases/Payments	114,754
Liabilities, Sales/Issuance	(402,769)
Liabilities, Transfers into Level 3	0	0
Liabilities, Transfers out of Level 3	0	0
Liabilities, Ending Balance	(1,216,542)	(911,256)
Change in unrealized gains (losses), liabilities	55,600	1,200
Financial Derivatives - Liabilities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gains (losses), liabilities		100
Senior notes
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	0
Liabilities, Accreted Discounts/Amortized Premiums	0
Liabilities, Realized Gain/(Loss)	0
Liabilities, Change In Net Unrealized Gain/(Loss)	0
Liabilities Purchases/Payments	0
Liabilities, Sales/Issuance	(210,000)
Liabilities, Transfers into Level 3	0
Liabilities, Transfers out of Level 3	0
Liabilities, Ending Balance	(210,000)
Securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	(4,300)	3,200
Securities | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	9,710	11,663
Assets, Accreted Discounts/Amortized Premiums	(573)	(1,121)
Assets, Realized Gain (Loss)	362	(26)
Assets, Change In Net Unrealized Gain/(Loss)	(1,211)	(136)
Assets, Purchases	399	1,814
Assets, Sales	(514)	0
Assets, Transfers into Level 3	1,500	5,857
Assets, Transfers out of Level 3	(1,052)	(649)
Assets, Ending Balance	8,621	17,402
Securities | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	134,888	127,838
Assets, Accreted Discounts/Amortized Premiums	479	679
Assets, Realized Gain (Loss)	(126)	(226)
Assets, Change In Net Unrealized Gain/(Loss)	(2,391)	(557)
Assets, Purchases	3,401	20,742
Assets, Sales	(6,688)	(18,639)
Assets, Transfers into Level 3	5,998	1,998
Assets, Transfers out of Level 3	(18,785)	(4,506)
Assets, Ending Balance	116,776	127,329
Securities | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	13,134	63,148
Assets, Accreted Discounts/Amortized Premiums	41	218
Assets, Realized Gain (Loss)	1,143	2,082
Assets, Change In Net Unrealized Gain/(Loss)	(747)	2,525
Assets, Purchases	3,101	0
Assets, Sales	(2,234)	(39,705)
Assets, Transfers into Level 3	2,926	0
Assets, Transfers out of Level 3	7,838	10,974
Assets, Ending Balance	9,526	17,294
Securities | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	26,678	111,100
Assets, Accreted Discounts/Amortized Premiums	(716)	734
Assets, Realized Gain (Loss)	953	912
Assets, Change In Net Unrealized Gain/(Loss)	1,610	5,334
Assets, Purchases	0	1,812
Assets, Sales	(5,781)	(43,210)
Assets, Transfers into Level 3	2,876	1,511
Assets, Transfers out of Level 3	(2,796)	(40,608)
Assets, Ending Balance	22,824	37,585
Securities | Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	73,108	44,925
Assets, Accreted Discounts/Amortized Premiums	(1,113)	(657)
Assets, Realized Gain (Loss)	(274)	33
Assets, Change In Net Unrealized Gain/(Loss)	(2,023)	(485)
Assets, Purchases	18,792	24,334
Assets, Sales	(11,986)	(8,677)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	76,504	59,473
Securities | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	5,198	4,082
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	1,535	180
Assets, Change In Net Unrealized Gain/(Loss)	(1,508)	5
Assets, Purchases	1,728	1,027
Assets, Sales	(6,453)	(533)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	500	4,761
Securities | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	7,556	1,590
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	1,625	(385)
Assets, Change In Net Unrealized Gain/(Loss)	(829)	604
Assets, Purchases	4,127	2,311
Assets, Sales	(2,638)	0
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	9,841	4,120
Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	(70,300)	2,000
Loans | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	2,016,228	1,187,069
Assets, Accreted Discounts/Amortized Premiums	(4,467)	(2,470)
Assets, Realized Gain (Loss)	1,511	194
Assets, Change In Net Unrealized Gain/(Loss)	(70,512)	2,214
Assets, Purchases	723,095	222,081
Assets, Sales	(232,848)	(128,451)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	2,433,007	1,280,637
Loans | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	326,197	213,031
Assets, Accreted Discounts/Amortized Premiums	0	9
Assets, Realized Gain (Loss)	10	436
Assets, Change In Net Unrealized Gain/(Loss)	164	(263)
Assets, Purchases	267,642	57,166
Assets, Sales	(164,059)	(34,431)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	429,954	235,948
Loans | Consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	62,365	47,525
Assets, Accreted Discounts/Amortized Premiums	(2,139)	(1,843)
Assets, Realized Gain (Loss)	(180)	(1,265)
Assets, Change In Net Unrealized Gain/(Loss)	(466)	327
Assets, Purchases	10,946	16,749
Assets, Sales	(60,648)	(8,788)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	9,878	52,705
Loans | Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	10,531	5,855
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	0	0
Assets, Change In Net Unrealized Gain/(Loss)	0	0
Assets, Purchases	1,650	7,371
Assets, Sales	(393)	0
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	11,788	13,226
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	195,643	141,620
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	878	128
Assets, Change In Net Unrealized Gain/(Loss)	(6,384)	6,507
Assets, Purchases	139,208	12,870
Assets, Sales	(110,042)	(13,441)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	219,303	147,684
Change in unrealized gain (loss), assets	(7,800)	6,300
Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	1	(33)
Financial Derivatives - Assets | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		9
Assets, Accreted Discounts/Amortized Premiums		0
Assets, Realized Gain (Loss)		141
Assets, Change In Net Unrealized Gain/(Loss)		(6)
Assets, Purchases		0
Assets, Sales		(142)
Assets, Transfers into Level 3		0
Assets, Transfers out of Level 3		0
Assets, Ending Balance		2
Financial Derivatives - Assets | Credit default swaps on asset-backed securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	303	347
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	(4)	26
Assets, Change In Net Unrealized Gain/(Loss)	1	(27)
Assets, Purchases	4	4
Assets, Sales	0	(30)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	$ 304	320
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Payments		500
Liabilities, Issuances		0
Liabilities, Begining Balance		(484)
Liabilities, Accreted Discounts/Amortized Premiums		0
Liabilities, Realized Gain/(Loss)		(500)
Liabilities, Change In Net Unrealized Gain/(Loss)		146
Liabilities, Transfers into Level 3		0
Liabilities, Transfers out of Level 3		0
Liabilities, Ending Balance		$ (338)