Valuation (Significant Unobservable Inputs Rollforward) (Details) - Level 3 - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 1,094,816
Assets, Accreted Discounts/Amortized Premiums	(40,792)
Assets, Realized Gain (Loss)	(3,380)
Assets, Change In Net Unrealized Gain/(Loss)	18,989
Assets, Purchases	1,229,160
Assets, Sales	(603,883)
Assets, Transfers into Level 3	35,567
Assets, Transfers out of Level 3	(9,202)
Assets, Ending Balance	1,721,275	$ 1,094,816
Liabilities, Begining Balance	(297,948)	(125,105)
Liabilities, Accreted Discounts/Amortized Premiums	0	0
Liabilities, Realized Gain/(Loss)	(15)	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(938)	758
Liabilities Purchases/Payments	182,306	49,731
Liabilities, Sales/Issuance	(478,237)	223,332
Liabilities, Transfers into Level 3	0	0
Liabilities, Transfers out of Level 3	0	0
Liabilities, Ending Balance	(594,832)	(297,948)
Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	(297,948)	(125,105)
Liabilities, Accreted Discounts/Amortized Premiums	0	0
Liabilities, Realized Gain/(Loss)	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(502)	758
Liabilities, Payments	182,291	49,731
Liabilities, Issuances	(478,237)	(223,332)
Liabilities, Transfers into Level 3	0	0
Liabilities, Transfers out of Level 3	0	0
Liabilities, Ending Balance	(594,396)	(297,948)
Change in unrealized gains (losses), liabilities	100	800
Financial Derivatives - Liabilities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gains (losses), liabilities	(400)
Securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	2,400
Securities | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	7,293
Assets, Accreted Discounts/Amortized Premiums	(3,464)
Assets, Realized Gain (Loss)	(1,787)
Assets, Change In Net Unrealized Gain/(Loss)	808
Assets, Purchases	13,818
Assets, Sales	(1,306)
Assets, Transfers into Level 3	5,370
Assets, Transfers out of Level 3	(828)
Assets, Ending Balance	19,904	7,293
Securities | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	91,291
Assets, Accreted Discounts/Amortized Premiums	270
Assets, Realized Gain (Loss)	5,636
Assets, Change In Net Unrealized Gain/(Loss)	(3,654)
Assets, Purchases	21,512
Assets, Sales	(33,664)
Assets, Transfers into Level 3	15,354
Assets, Transfers out of Level 3	(7,164)
Assets, Ending Balance	89,581	91,291
Securities | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	803
Assets, Accreted Discounts/Amortized Premiums	16
Assets, Realized Gain (Loss)	180
Assets, Change In Net Unrealized Gain/(Loss)	(246)
Assets, Purchases	31,464
Assets, Sales	(5,271)
Assets, Transfers into Level 3	2,859
Assets, Transfers out of Level 3	0
Assets, Ending Balance	29,805	803
Securities | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	14,915
Assets, Accreted Discounts/Amortized Premiums	(268)
Assets, Realized Gain (Loss)	(3,190)
Assets, Change In Net Unrealized Gain/(Loss)	2,329
Assets, Purchases	25,531
Assets, Sales	(5,112)
Assets, Transfers into Level 3	11,984
Assets, Transfers out of Level 3	(1,210)
Assets, Ending Balance	44,979	14,915
Securities | Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	22,800
Assets, Accreted Discounts/Amortized Premiums	(2,520)
Assets, Realized Gain (Loss)	(891)
Assets, Change In Net Unrealized Gain/(Loss)	873
Assets, Purchases	42,137
Assets, Sales	(13,789)
Assets, Transfers into Level 3	0
Assets, Transfers out of Level 3	0
Assets, Ending Balance	48,610	22,800
Securities | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	6,318
Assets, Accreted Discounts/Amortized Premiums	22
Assets, Realized Gain (Loss)	(1,341)
Assets, Change In Net Unrealized Gain/(Loss)	188
Assets, Purchases	11,024
Assets, Sales	(15,098)
Assets, Transfers into Level 3	0
Assets, Transfers out of Level 3	0
Assets, Ending Balance	1,113	6,318
Securities | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,534
Assets, Accreted Discounts/Amortized Premiums	0
Assets, Realized Gain (Loss)	(1,807)
Assets, Change In Net Unrealized Gain/(Loss)	205
Assets, Purchases	1,462
Assets, Sales	0
Assets, Transfers into Level 3	0
Assets, Transfers out of Level 3	0
Assets, Ending Balance	1,394	1,534
Investments
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,124,606	658,448
Assets, Accreted Discounts/Amortized Premiums		(32,013)
Assets, Realized Gain (Loss)		16,205
Assets, Change In Net Unrealized Gain/(Loss)		3,821
Assets, Purchases		937,544
Assets, Sales		(438,411)
Assets, Transfers into Level 3		13,649
Assets, Transfers out of Level 3		(34,637)
Assets, Ending Balance		1,124,606
Change in unrealized gain (loss), assets		5,300
Investments | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	7,293	6,173
Assets, Accreted Discounts/Amortized Premiums		(2,233)
Assets, Realized Gain (Loss)		(10)
Assets, Change In Net Unrealized Gain/(Loss)		175
Assets, Purchases		2,753
Assets, Sales		(1,169)
Assets, Transfers into Level 3		2,616
Assets, Transfers out of Level 3		(1,012)
Assets, Ending Balance		7,293
Investments | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	91,291	101,297
Assets, Accreted Discounts/Amortized Premiums		383
Assets, Realized Gain (Loss)		1,838
Assets, Change In Net Unrealized Gain/(Loss)		(2,135)
Assets, Purchases		75,685
Assets, Sales		(78,487)
Assets, Transfers into Level 3		7,074
Assets, Transfers out of Level 3		(14,364)
Assets, Ending Balance		91,291
Investments | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	803	12,347
Assets, Accreted Discounts/Amortized Premiums		(243)
Assets, Realized Gain (Loss)		2,229
Assets, Change In Net Unrealized Gain/(Loss)		2,120
Assets, Purchases		1,481
Assets, Sales		(16,896)
Assets, Transfers into Level 3		0
Assets, Transfers out of Level 3		(235)
Assets, Ending Balance		803
Investments | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	14,915	24,911
Assets, Accreted Discounts/Amortized Premiums		(351)
Assets, Realized Gain (Loss)		317
Assets, Change In Net Unrealized Gain/(Loss)		(2,268)
Assets, Purchases		33,549
Assets, Sales		(33,115)
Assets, Transfers into Level 3		3,959
Assets, Transfers out of Level 3		(12,087)
Assets, Ending Balance		14,915
Investments | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	6,318	23,947
Assets, Accreted Discounts/Amortized Premiums		56
Assets, Realized Gain (Loss)		241
Assets, Change In Net Unrealized Gain/(Loss)		(964)
Assets, Purchases		7,665
Assets, Sales		(17,688)
Assets, Transfers into Level 3		0
Assets, Transfers out of Level 3		(6,939)
Assets, Ending Balance		6,318
Investments | Secured notes
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	10,917	0
Assets, Accreted Discounts/Amortized Premiums		870
Assets, Realized Gain (Loss)		0
Assets, Change In Net Unrealized Gain/(Loss)		(1,221)
Assets, Purchases		11,268
Assets, Sales		0
Assets, Transfers into Level 3		0
Assets, Transfers out of Level 3		0
Assets, Ending Balance		10,917
Investments | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	43,793	37,465
Assets, Accreted Discounts/Amortized Premiums		0
Assets, Realized Gain (Loss)		1,671
Assets, Change In Net Unrealized Gain/(Loss)		8,299
Assets, Purchases		12,708
Assets, Sales		(16,350)
Assets, Transfers into Level 3		0
Assets, Transfers out of Level 3		0
Assets, Ending Balance		43,793
Investments | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	496,830	182,472
Assets, Accreted Discounts/Amortized Premiums		(1,965)
Assets, Realized Gain (Loss)		1,011
Assets, Change In Net Unrealized Gain/(Loss)		(34)
Assets, Purchases		402,235
Assets, Sales		(86,889)
Assets, Transfers into Level 3		0
Assets, Transfers out of Level 3		0
Assets, Ending Balance		496,830
Investments | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		108,301
Assets, Accreted Discounts/Amortized Premiums		790
Assets, Realized Gain (Loss)		1,146
Assets, Change In Net Unrealized Gain/(Loss)		1,944
Assets, Purchases		149,053
Assets, Sales		(50,049)
Assets, Transfers into Level 3		0
Assets, Transfers out of Level 3		0
Investments | Consumer loans and asset-backed securities backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	206,761	135,258
Assets, Accreted Discounts/Amortized Premiums		(29,320)
Assets, Realized Gain (Loss)		8,415
Assets, Change In Net Unrealized Gain/(Loss)		(1,092)
Assets, Purchases		228,354
Assets, Sales		(134,854)
Assets, Transfers into Level 3		0
Assets, Transfers out of Level 3		0
Assets, Ending Balance		206,761
Investments | Real estate owned
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	34,500	26,277
Assets, Accreted Discounts/Amortized Premiums		0
Assets, Realized Gain (Loss)		(653)
Assets, Change In Net Unrealized Gain/(Loss)		(1,003)
Assets, Purchases		12,793
Assets, Sales		(2,914)
Assets, Transfers into Level 3		0
Assets, Transfers out of Level 3		0
Assets, Ending Balance		34,500
Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	11,500
Loans | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	496,830
Assets, Accreted Discounts/Amortized Premiums	(6,081)
Assets, Realized Gain (Loss)	1,466
Assets, Change In Net Unrealized Gain/(Loss)	8,800
Assets, Purchases	661,813
Assets, Sales	(230,625)
Assets, Transfers into Level 3	0
Assets, Transfers out of Level 3	0
Assets, Ending Balance	932,203	496,830
Loans | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	195,301
Assets, Accreted Discounts/Amortized Premiums	(282)
Assets, Realized Gain (Loss)	2,412
Assets, Change In Net Unrealized Gain/(Loss)	(2,083)
Assets, Purchases	175,689
Assets, Sales	(96,278)
Assets, Transfers into Level 3	0
Assets, Transfers out of Level 3	0
Assets, Ending Balance	274,759	195,301
Loans | Consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	183,961
Assets, Accreted Discounts/Amortized Premiums	(28,521)
Assets, Realized Gain (Loss)	(6,291)
Assets, Change In Net Unrealized Gain/(Loss)	3,000
Assets, Purchases	183,994
Assets, Sales	(149,189)
Assets, Transfers into Level 3	0
Assets, Transfers out of Level 3	0
Assets, Ending Balance	186,954	183,961
Loans | Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	0
Assets, Accreted Discounts/Amortized Premiums	36
Assets, Realized Gain (Loss)	0
Assets, Change In Net Unrealized Gain/(Loss)	(36)
Assets, Purchases	18,510
Assets, Sales	0
Assets, Transfers into Level 3	0
Assets, Transfers out of Level 3	0
Assets, Ending Balance	18,510	0
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	72,298
Assets, Accreted Discounts/Amortized Premiums	0
Assets, Realized Gain (Loss)	1,545
Assets, Change In Net Unrealized Gain/(Loss)	8,664
Assets, Purchases	42,173
Assets, Sales	(52,830)
Assets, Transfers into Level 3	0
Assets, Transfers out of Level 3	0
Assets, Ending Balance	71,850	72,298
Change in unrealized gain (loss), assets	5,200
Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,472	3,140
Assets, Accreted Discounts/Amortized Premiums		0
Assets, Realized Gain (Loss)		(687)
Assets, Change In Net Unrealized Gain/(Loss)		715
Assets, Purchases		102
Assets, Sales		(1,798)
Assets, Transfers into Level 3		0
Assets, Transfers out of Level 3		0
Assets, Ending Balance		1,472
Change in unrealized gain (loss), assets	100	(600)
Financial Derivatives - Assets | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	0
Assets, Accreted Discounts/Amortized Premiums	0
Assets, Realized Gain (Loss)	160
Assets, Change In Net Unrealized Gain/(Loss)	620
Assets, Purchases	0
Assets, Sales	(160)
Assets, Transfers into Level 3	0
Assets, Transfers out of Level 3	0
Assets, Ending Balance	620	0
Financial Derivatives - Assets | Credit default swaps on asset-backed securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,472	3,140
Assets, Accreted Discounts/Amortized Premiums	0	0
Assets, Realized Gain (Loss)	528	(687)
Assets, Change In Net Unrealized Gain/(Loss)	(479)	715
Assets, Purchases	33	102
Assets, Sales	(561)	(1,798)
Assets, Transfers into Level 3	0	0
Assets, Transfers out of Level 3	0	0
Assets, Ending Balance	993	1,472
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	0
Liabilities, Accreted Discounts/Amortized Premiums	0
Liabilities, Realized Gain/(Loss)	(15)
Liabilities, Change In Net Unrealized Gain/(Loss)	(436)
Liabilities, Payments	15
Liabilities, Issuances	0
Liabilities, Transfers into Level 3	0
Liabilities, Transfers out of Level 3	0
Liabilities, Ending Balance	(436)	0
Previous Accounting Guidance [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,126,078	661,588
Assets, Accreted Discounts/Amortized Premiums		(32,013)
Assets, Realized Gain (Loss)		15,518
Assets, Change In Net Unrealized Gain/(Loss)		4,536
Assets, Purchases		937,646
Assets, Sales		(440,209)
Assets, Transfers into Level 3		13,649
Assets, Transfers out of Level 3		(34,637)
Assets, Ending Balance		1,126,078
Previous Accounting Guidance [Member] | Investments | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 211,185
Assets, Ending Balance		$ 211,185