Valuation (Significant Unobservable Inputs Rollforward) (Details) - Level 3 - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 1,721,275	$ 1,094,816
Assets, Accreted Discounts/Amortized Premiums	(39,237)	(40,792)
Assets, Realized Gain (Loss)	(15,898)	(3,380)
Assets, Change In Net Unrealized Gain/(Loss)	36,777	18,989
Assets, Purchases	1,131,307	1,229,160
Assets, Sales	(945,519)	(603,883)
Assets, Transfers into Level 3	92,143	35,567
Assets, Transfers out of Level 3	(21,046)	(9,202)
Assets, Ending Balance	1,959,802	1,721,275	$ 1,094,816
Liabilities, Begining Balance	(594,832)	(297,948)	(125,105)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0
Liabilities, Realized Gain/(Loss)	(551)	(15)	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(9,624)	(938)	758
Liabilities Purchases/Payments	306,420	182,306	49,731
Liabilities, Sales/Issuance	(456,818)	(478,237)	223,332
Liabilities, Transfers into Level 3	0	0	0
Liabilities, Transfers out of Level 3	0	0	0
Liabilities, Ending Balance	(755,405)	(594,832)	(297,948)
Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	(594,396)	(297,948)	(125,105)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0
Liabilities, Realized Gain/(Loss)	0	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(9,576)	(502)	758
Liabilities, Payments	305,828	182,291	49,731
Liabilities, Issuances	(456,777)	(478,237)	(223,332)
Liabilities, Transfers into Level 3	0	0	0
Liabilities, Transfers out of Level 3	0	0	0
Liabilities, Ending Balance	(754,921)	(594,396)	(297,948)
Change in unrealized gains (losses), liabilities	(9,600)	100	800
Financial Derivatives - Liabilities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gains (losses), liabilities	(500)	(400)
Securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	(33,300)	2,400
Securities | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	19,904	7,293
Assets, Accreted Discounts/Amortized Premiums	(7,903)	(3,464)
Assets, Realized Gain (Loss)	722	(1,787)
Assets, Change In Net Unrealized Gain/(Loss)	3,175	808
Assets, Purchases	8,307	13,818
Assets, Sales	(5,046)	(1,306)
Assets, Transfers into Level 3	1,083	5,370
Assets, Transfers out of Level 3	(8,579)	(828)
Assets, Ending Balance	11,663	19,904	7,293
Securities | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	89,581	91,291
Assets, Accreted Discounts/Amortized Premiums	1,557	270
Assets, Realized Gain (Loss)	1,009	5,636
Assets, Change In Net Unrealized Gain/(Loss)	(1,283)	(3,654)
Assets, Purchases	64,362	21,512
Assets, Sales	(40,841)	(33,664)
Assets, Transfers into Level 3	17,425	15,354
Assets, Transfers out of Level 3	(3,972)	(7,164)
Assets, Ending Balance	127,838	89,581	91,291
Securities | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	29,805	803
Assets, Accreted Discounts/Amortized Premiums	813	16
Assets, Realized Gain (Loss)	62	180
Assets, Change In Net Unrealized Gain/(Loss)	(2,477)	(246)
Assets, Purchases	52,915	31,464
Assets, Sales	(38,553)	(5,271)
Assets, Transfers into Level 3	20,583	2,859
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	63,148	29,805	803
Securities | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	44,979	14,915
Assets, Accreted Discounts/Amortized Premiums	2,185	(268)
Assets, Realized Gain (Loss)	(8,862)	(3,190)
Assets, Change In Net Unrealized Gain/(Loss)	(13,132)	2,329
Assets, Purchases	48,120	25,531
Assets, Sales	(6,747)	(5,112)
Assets, Transfers into Level 3	53,052	11,984
Assets, Transfers out of Level 3	(8,495)	(1,210)
Assets, Ending Balance	111,100	44,979	14,915
Securities | Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	48,610	22,800
Assets, Accreted Discounts/Amortized Premiums	(4,986)	(2,520)
Assets, Realized Gain (Loss)	(138)	(891)
Assets, Change In Net Unrealized Gain/(Loss)	(1,245)	873
Assets, Purchases	30,899	42,137
Assets, Sales	(28,215)	(13,789)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	44,925	48,610	22,800
Securities | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,113	6,318
Assets, Accreted Discounts/Amortized Premiums	0	22
Assets, Realized Gain (Loss)	914	(1,341)
Assets, Change In Net Unrealized Gain/(Loss)	1,068	188
Assets, Purchases	5,668	11,024
Assets, Sales	(4,681)	(15,098)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	4,082	1,113	6,318
Securities | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,394	1,534
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	7	(1,807)
Assets, Change In Net Unrealized Gain/(Loss)	(165)	205
Assets, Purchases	366	1,462
Assets, Sales	(12)	0
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	1,590	1,394	1,534
Investments
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		1,124,606	658,448
Assets, Accreted Discounts/Amortized Premiums			(32,013)
Assets, Realized Gain (Loss)			16,205
Assets, Change In Net Unrealized Gain/(Loss)			3,821
Assets, Purchases			937,544
Assets, Sales			(438,411)
Assets, Transfers into Level 3			13,649
Assets, Transfers out of Level 3			(34,637)
Assets, Ending Balance			1,124,606
Change in unrealized gain (loss), assets			5,300
Investments | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		7,293	6,173
Assets, Accreted Discounts/Amortized Premiums			(2,233)
Assets, Realized Gain (Loss)			(10)
Assets, Change In Net Unrealized Gain/(Loss)			175
Assets, Purchases			2,753
Assets, Sales			(1,169)
Assets, Transfers into Level 3			2,616
Assets, Transfers out of Level 3			(1,012)
Assets, Ending Balance			7,293
Investments | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		91,291	101,297
Assets, Accreted Discounts/Amortized Premiums			383
Assets, Realized Gain (Loss)			1,838
Assets, Change In Net Unrealized Gain/(Loss)			(2,135)
Assets, Purchases			75,685
Assets, Sales			(78,487)
Assets, Transfers into Level 3			7,074
Assets, Transfers out of Level 3			(14,364)
Assets, Ending Balance			91,291
Investments | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		803	12,347
Assets, Accreted Discounts/Amortized Premiums			(243)
Assets, Realized Gain (Loss)			2,229
Assets, Change In Net Unrealized Gain/(Loss)			2,120
Assets, Purchases			1,481
Assets, Sales			(16,896)
Assets, Transfers into Level 3			0
Assets, Transfers out of Level 3			(235)
Assets, Ending Balance			803
Investments | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		14,915	24,911
Assets, Accreted Discounts/Amortized Premiums			(351)
Assets, Realized Gain (Loss)			317
Assets, Change In Net Unrealized Gain/(Loss)			(2,268)
Assets, Purchases			33,549
Assets, Sales			(33,115)
Assets, Transfers into Level 3			3,959
Assets, Transfers out of Level 3			(12,087)
Assets, Ending Balance			14,915
Investments | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		6,318	23,947
Assets, Accreted Discounts/Amortized Premiums			56
Assets, Realized Gain (Loss)			241
Assets, Change In Net Unrealized Gain/(Loss)			(964)
Assets, Purchases			7,665
Assets, Sales			(17,688)
Assets, Transfers into Level 3			0
Assets, Transfers out of Level 3			(6,939)
Assets, Ending Balance			6,318
Investments | Secured notes
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		10,917	0
Assets, Accreted Discounts/Amortized Premiums			870
Assets, Realized Gain (Loss)			0
Assets, Change In Net Unrealized Gain/(Loss)			(1,221)
Assets, Purchases			11,268
Assets, Sales			0
Assets, Transfers into Level 3			0
Assets, Transfers out of Level 3			0
Assets, Ending Balance			10,917
Investments | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		43,793	37,465
Assets, Accreted Discounts/Amortized Premiums			0
Assets, Realized Gain (Loss)			1,671
Assets, Change In Net Unrealized Gain/(Loss)			8,299
Assets, Purchases			12,708
Assets, Sales			(16,350)
Assets, Transfers into Level 3			0
Assets, Transfers out of Level 3			0
Assets, Ending Balance			43,793
Investments | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		496,830	182,472
Assets, Accreted Discounts/Amortized Premiums			(1,965)
Assets, Realized Gain (Loss)			1,011
Assets, Change In Net Unrealized Gain/(Loss)			(34)
Assets, Purchases			402,235
Assets, Sales			(86,889)
Assets, Transfers into Level 3			0
Assets, Transfers out of Level 3			0
Assets, Ending Balance			496,830
Investments | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance			108,301
Assets, Accreted Discounts/Amortized Premiums			790
Assets, Realized Gain (Loss)			1,146
Assets, Change In Net Unrealized Gain/(Loss)			1,944
Assets, Purchases			149,053
Assets, Sales			(50,049)
Assets, Transfers into Level 3			0
Assets, Transfers out of Level 3			0
Investments | Consumer loans and asset-backed securities backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		206,761	135,258
Assets, Accreted Discounts/Amortized Premiums			(29,320)
Assets, Realized Gain (Loss)			8,415
Assets, Change In Net Unrealized Gain/(Loss)			(1,092)
Assets, Purchases			228,354
Assets, Sales			(134,854)
Assets, Transfers into Level 3			0
Assets, Transfers out of Level 3			0
Assets, Ending Balance			206,761
Investments | Real estate owned
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		34,500	26,277
Assets, Accreted Discounts/Amortized Premiums			0
Assets, Realized Gain (Loss)			(653)
Assets, Change In Net Unrealized Gain/(Loss)			(1,003)
Assets, Purchases			12,793
Assets, Sales			(2,914)
Assets, Transfers into Level 3			0
Assets, Transfers out of Level 3			0
Assets, Ending Balance			34,500
Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	8,600	11,500
Loans | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	932,203	496,830
Assets, Accreted Discounts/Amortized Premiums	(6,445)	(6,081)
Assets, Realized Gain (Loss)	(165)	1,466
Assets, Change In Net Unrealized Gain/(Loss)	11,593	8,800
Assets, Purchases	594,397	661,813
Assets, Sales	(344,514)	(230,625)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	1,187,069	932,203	496,830
Loans | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	274,759	195,301
Assets, Accreted Discounts/Amortized Premiums	128	(282)
Assets, Realized Gain (Loss)	135	2,412
Assets, Change In Net Unrealized Gain/(Loss)	(166)	(2,083)
Assets, Purchases	121,844	175,689
Assets, Sales	(183,669)	(96,278)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	213,031	274,759	195,301
Loans | Consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	186,954	183,961
Assets, Accreted Discounts/Amortized Premiums	(24,586)	(28,521)
Assets, Realized Gain (Loss)	(4,843)	(6,291)
Assets, Change In Net Unrealized Gain/(Loss)	(2,891)	3,000
Assets, Purchases	141,245	183,994
Assets, Sales	(248,354)	(149,189)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	47,525	186,954	183,961
Loans | Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	18,510	0
Assets, Accreted Discounts/Amortized Premiums	0	36
Assets, Realized Gain (Loss)	0	0
Assets, Change In Net Unrealized Gain/(Loss)	0	(36)
Assets, Purchases	1,445	18,510
Assets, Sales	(14,100)	0
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	5,855	18,510	0
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	71,850	72,298
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	424	1,545
Assets, Change In Net Unrealized Gain/(Loss)	37,509	8,664
Assets, Purchases	61,589	42,173
Assets, Sales	(29,752)	(52,830)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	141,620	71,850	72,298
Change in unrealized gain (loss), assets	37,100	5,200
Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		1,472	3,140
Assets, Accreted Discounts/Amortized Premiums			0
Assets, Realized Gain (Loss)			(687)
Assets, Change In Net Unrealized Gain/(Loss)			715
Assets, Purchases			102
Assets, Sales			(1,798)
Assets, Transfers into Level 3			0
Assets, Transfers out of Level 3			0
Assets, Ending Balance			1,472
Change in unrealized gain (loss), assets	500	100	(600)
Financial Derivatives - Assets | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	620	0
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	288	160
Assets, Change In Net Unrealized Gain/(Loss)	(611)	620
Assets, Purchases	126	0
Assets, Sales	(414)	(160)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	9	620	0
Financial Derivatives - Assets | Credit default swaps on asset-backed securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	993	1,472	3,140
Assets, Accreted Discounts/Amortized Premiums	0	0	0
Assets, Realized Gain (Loss)	(5,451)	528	(687)
Assets, Change In Net Unrealized Gain/(Loss)	5,402	(479)	715
Assets, Purchases	24	33	102
Assets, Sales	(621)	(561)	(1,798)
Assets, Transfers into Level 3	0	0	0
Assets, Transfers out of Level 3	0	0	0
Assets, Ending Balance	347	993	1,472
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	(436)	0
Liabilities, Accreted Discounts/Amortized Premiums	0	0
Liabilities, Realized Gain/(Loss)	(551)	(15)
Liabilities, Change In Net Unrealized Gain/(Loss)	(48)	(436)
Liabilities, Payments	592	15
Liabilities, Issuances	(41)	0
Liabilities, Transfers into Level 3	0	0
Liabilities, Transfers out of Level 3	0	0
Liabilities, Ending Balance	$ (484)	(436)	0
Previous Accounting Guidance [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		1,126,078	661,588
Assets, Accreted Discounts/Amortized Premiums			(32,013)
Assets, Realized Gain (Loss)			15,518
Assets, Change In Net Unrealized Gain/(Loss)			4,536
Assets, Purchases			937,646
Assets, Sales			(440,209)
Assets, Transfers into Level 3			13,649
Assets, Transfers out of Level 3			(34,637)
Assets, Ending Balance			1,126,078
Previous Accounting Guidance [Member] | Investments | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		$ 211,185
Assets, Ending Balance			$ 211,185