Valuation (Significant Unobservable Inputs Rollforward) (Details) - Level 3 - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2019	Sep. 30, 2018	Sep. 30, 2019	Sep. 30, 2018
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	$ 1,324,591	$ 816,506	$ 1,094,815	$ 661,588
Assets, Accreted Discounts/Amortized Premiums	(10,822)	(8,816)	(29,952)	(23,104)
Assets, Realized Gain (Loss)	5,702	7,823	3,262	16,398
Assets, Change In Net Unrealized Gain/(Loss)	1,047	(8,223)	13,268	2,799
Assets, Purchases	330,413	289,205	814,294	674,353
Assets, Sales	(173,334)	(112,163)	(433,042)	(322,881)
Assets, Transfers into Level 3	22,484	10,107	38,490	11,120
Assets, Transfers out of Level 3	(12,357)	(17,260)	(13,411)	(43,094)
Assets, Ending Balance	1,487,724	977,179	1,487,724	977,179
Liabilities, Begining Balance	(475,816)	(101,100)	(297,948)	(125,105)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	48	0	48	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(286)	(358)	(246)	840
Liabilities Purchases/Payments	37,259	11,889	78,887	34,696
Liabilities, Sales/Issuance	(48)	0	(219,584)	0
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(438,843)	(89,569)	(438,843)	(89,569)
Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	(475,816)	(101,100)	(297,948)	(125,105)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	0	0	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(72)	(358)	(32)	840
Liabilities, Payments	37,259	11,889	78,887	34,696
Liabilities, Issuances	0	0	(219,536)	0
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(438,629)	(89,569)	(438,629)	(89,569)
Change in unrealized gains (losses), liabilities	(100)	(400)	(32)	800
Financial Derivatives - Liabilities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gains (losses), liabilities	(200)		(200)
Securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	(4,700)		(400)
Securities | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	11,034		7,293
Assets, Accreted Discounts/Amortized Premiums	(1,164)		(2,228)
Assets, Realized Gain (Loss)	(621)		(1,228)
Assets, Change In Net Unrealized Gain/(Loss)	696		1,430
Assets, Purchases	13,254		14,854
Assets, Sales	0		(463)
Assets, Transfers into Level 3	737		4,708
Assets, Transfers out of Level 3	(437)		(867)
Assets, Ending Balance	23,499		23,499
Securities | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	96,790		91,291
Assets, Accreted Discounts/Amortized Premiums	(121)		104
Assets, Realized Gain (Loss)	7,836		7,187
Assets, Change In Net Unrealized Gain/(Loss)	(6,457)		(4,482)
Assets, Purchases	6,483		10,239
Assets, Sales	(21,457)		(27,514)
Assets, Transfers into Level 3	5,808		14,633
Assets, Transfers out of Level 3	(7,400)		(9,976)
Assets, Ending Balance	81,482		81,482
Securities | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	6,278		803
Assets, Accreted Discounts/Amortized Premiums	25		(16)
Assets, Realized Gain (Loss)	374		76
Assets, Change In Net Unrealized Gain/(Loss)	(302)		(73)
Assets, Purchases	6,815		7,937
Assets, Sales	(1,474)		(221)
Assets, Transfers into Level 3	652		3,862
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	12,368		12,368
Securities | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	17,222		14,915
Assets, Accreted Discounts/Amortized Premiums	(184)		(670)
Assets, Realized Gain (Loss)	1,158		(536)
Assets, Change In Net Unrealized Gain/(Loss)	184		1,889
Assets, Purchases	0		816
Assets, Sales	(1,139)		(1,125)
Assets, Transfers into Level 3	15,287		15,287
Assets, Transfers out of Level 3	(4,520)		(2,568)
Assets, Ending Balance	28,008		28,008
Securities | Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	25,019		22,800
Assets, Accreted Discounts/Amortized Premiums	(611)		(1,580)
Assets, Realized Gain (Loss)	(100)		(765)
Assets, Change In Net Unrealized Gain/(Loss)	(320)		537
Assets, Purchases	18,638		28,189
Assets, Sales	(3,310)		(9,865)
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	39,316		39,316
Securities | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	4,081		6,318
Assets, Accreted Discounts/Amortized Premiums	0		22
Assets, Realized Gain (Loss)	(583)		(928)
Assets, Change In Net Unrealized Gain/(Loss)	142		65
Assets, Purchases	6,425		9,257
Assets, Sales	(6,883)		(11,552)
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	3,182		3,182
Securities | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,791		1,534
Assets, Accreted Discounts/Amortized Premiums	0		0
Assets, Realized Gain (Loss)	(768)		(910)
Assets, Change In Net Unrealized Gain/(Loss)	(61)		337
Assets, Purchases	1,316		1,317
Assets, Sales	0		0
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	2,278		2,278
Investments
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		813,915		658,448
Assets, Accreted Discounts/Amortized Premiums		(8,816)		(23,104)
Assets, Realized Gain (Loss)		8,809		17,137
Assets, Change In Net Unrealized Gain/(Loss)		(9,530)		2,040
Assets, Purchases		289,178		674,281
Assets, Sales		(110,739)		(321,164)
Assets, Transfers into Level 3		10,107		11,120
Assets, Transfers out of Level 3		(17,260)		(43,094)
Assets, Ending Balance		975,664		975,664
Change in unrealized gain (loss), assets		(2,400)		1,800
Investments | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		5,889		6,173
Assets, Accreted Discounts/Amortized Premiums		(509)		(1,711)
Assets, Realized Gain (Loss)		(24)		34
Assets, Change In Net Unrealized Gain/(Loss)		(67)		206
Assets, Purchases		773		2,751
Assets, Sales		(209)		(1,166)
Assets, Transfers into Level 3		3,981		3,981
Assets, Transfers out of Level 3		(1,055)		(1,489)
Assets, Ending Balance		8,779		8,779
Investments | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		96,396		101,297
Assets, Accreted Discounts/Amortized Premiums		(206)		1
Assets, Realized Gain (Loss)		335		2,836
Assets, Change In Net Unrealized Gain/(Loss)		(1,181)		(2,392)
Assets, Purchases		48,860		76,816
Assets, Sales		(16,726)		(43,723)
Assets, Transfers into Level 3		3,942		7,139
Assets, Transfers out of Level 3		(10,146)		(20,700)
Assets, Ending Balance		121,274		121,274
Investments | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		8,761		12,347
Assets, Accreted Discounts/Amortized Premiums		4		(225)
Assets, Realized Gain (Loss)		198		1,815
Assets, Change In Net Unrealized Gain/(Loss)		494		2,111
Assets, Purchases		0		1,480
Assets, Sales		(2,415)		(16,305)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		(6,059)		(240)
Assets, Ending Balance		983		983
Investments | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		6,109		24,911
Assets, Accreted Discounts/Amortized Premiums		302		345
Assets, Realized Gain (Loss)		(25)		826
Assets, Change In Net Unrealized Gain/(Loss)		(491)		(991)
Assets, Purchases		24,115		47,837
Assets, Sales		(3,183)		(30,309)
Assets, Transfers into Level 3		2,184		0
Assets, Transfers out of Level 3		0		(13,608)
Assets, Ending Balance		29,011		29,011
Investments | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		8,850		23,947
Assets, Accreted Discounts/Amortized Premiums		13		0
Assets, Realized Gain (Loss)		87		235
Assets, Change In Net Unrealized Gain/(Loss)		(280)		(709)
Assets, Purchases		6,780		7,684
Assets, Sales		(7,769)		(16,419)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		(7,057)
Assets, Ending Balance		7,681		7,681
Investments | Secured notes
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		11,126		0
Assets, Accreted Discounts/Amortized Premiums		405		497
Assets, Realized Gain (Loss)		0		0
Assets, Change In Net Unrealized Gain/(Loss)		(466)		(700)
Assets, Purchases		0		11,268
Assets, Sales		0		0
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		11,065		11,065
Investments | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		44,768		37,465
Assets, Accreted Discounts/Amortized Premiums		0		0
Assets, Realized Gain (Loss)		487		1,669
Assets, Change In Net Unrealized Gain/(Loss)		(377)		3,948
Assets, Purchases		0		9,078
Assets, Sales		(5,723)		(13,005)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		39,155		39,155
Investments | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		293,472		182,472
Assets, Accreted Discounts/Amortized Premiums		(513)		(1,692)
Assets, Realized Gain (Loss)		25		608
Assets, Change In Net Unrealized Gain/(Loss)		(400)		(1,319)
Assets, Purchases		117,101		266,925
Assets, Sales		(17,225)		(54,534)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		392,460		392,460
Investments | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		104,951		108,301
Assets, Accreted Discounts/Amortized Premiums		(300)		545
Assets, Realized Gain (Loss)		252		1,135
Assets, Change In Net Unrealized Gain/(Loss)		522		971
Assets, Purchases		36,571		60,691
Assets, Sales		(16,188)		(45,835)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		125,808		125,808
Investments | Consumer loans and asset-backed securities backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		199,254		135,258
Assets, Accreted Discounts/Amortized Premiums		(8,012)		(20,864)
Assets, Realized Gain (Loss)		7,555		8,506
Assets, Change In Net Unrealized Gain/(Loss)		(7,615)		(30)
Assets, Purchases		54,503		179,706
Assets, Sales		(41,181)		(98,072)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		204,504		204,504
Investments | Real estate owned
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		34,339		26,277
Assets, Accreted Discounts/Amortized Premiums		0		0
Assets, Realized Gain (Loss)		(81)		(527)
Assets, Change In Net Unrealized Gain/(Loss)		331		945
Assets, Purchases		475		10,045
Assets, Sales		(120)		(1,796)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		34,944		34,944
Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	4,100		10,200
Loans | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	663,880		496,829
Assets, Accreted Discounts/Amortized Premiums	(2,241)		(4,515)
Assets, Realized Gain (Loss)	(400)		1,554
Assets, Change In Net Unrealized Gain/(Loss)	3,559		7,717
Assets, Purchases	191,512		452,958
Assets, Sales	(58,582)		(156,815)
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	797,728		797,728
Loans | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	260,034		195,301
Assets, Accreted Discounts/Amortized Premiums	(52)		1,087
Assets, Realized Gain (Loss)	(1)		1,412
Assets, Change In Net Unrealized Gain/(Loss)	507		(1,844)
Assets, Purchases	32,426		128,839
Assets, Sales	(32,288)		(64,169)
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	260,626		260,626
Loans | Consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	162,609		183,961
Assets, Accreted Discounts/Amortized Premiums	(6,474)		(22,432)
Assets, Realized Gain (Loss)	(1,055)		(4,565)
Assets, Change In Net Unrealized Gain/(Loss)	28		2,726
Assets, Purchases	33,101		103,983
Assets, Sales	(36,510)		(111,974)
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	151,699		151,699
Loans | Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	5,000		0
Assets, Accreted Discounts/Amortized Premiums	0		0
Assets, Realized Gain (Loss)	0		0
Assets, Change In Net Unrealized Gain/(Loss)	0		0
Assets, Purchases	10,790		15,790
Assets, Sales	0		0
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	15,790		15,790
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	69,676		72,298
Assets, Accreted Discounts/Amortized Premiums	0		276
Assets, Realized Gain (Loss)	(139)		1,545
Assets, Change In Net Unrealized Gain/(Loss)	2,935		5,125
Assets, Purchases	9,643		40,097
Assets, Sales	(11,680)		(48,906)
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	70,435		70,435
Change in unrealized gain (loss), assets	3,300		2,400
Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		2,591		3,140
Assets, Accreted Discounts/Amortized Premiums		0		0
Assets, Realized Gain (Loss)		(986)		(739)
Assets, Change In Net Unrealized Gain/(Loss)		1,307		759
Assets, Purchases		27		72
Assets, Sales		(1,424)		(1,717)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		1,515		1,515
Change in unrealized gain (loss), assets	100	(100)	(200)	(600)
Financial Derivatives - Assets | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	87		0
Assets, Accreted Discounts/Amortized Premiums	0		0
Assets, Realized Gain (Loss)	(15)		1
Assets, Change In Net Unrealized Gain/(Loss)	148		235
Assets, Purchases	0		0
Assets, Sales	15		(1)
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	235		235
Financial Derivatives - Assets | Credit default swaps on asset-backed securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,090	2,591	1,472	3,140
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	16	(986)	419	(739)
Assets, Change In Net Unrealized Gain/(Loss)	(12)	1,307	(394)	759
Assets, Purchases	10	27	18	72
Assets, Sales	(26)	(1,424)	(437)	(1,717)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	1,078	$ 1,515	1,078	$ 1,515
Financial Derivatives - Liabilities | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	0		0
Liabilities, Accreted Discounts/Amortized Premiums	0		0
Liabilities, Realized Gain/(Loss)	48		48
Liabilities, Change In Net Unrealized Gain/(Loss)	(214)		(214)
Liabilities, Payments	0		0
Liabilities, Issuances	(48)		(48)
Liabilities, Transfers into Level 3	0		0
Liabilities, Transfers out of Level 3	0		0
Liabilities, Ending Balance	$ (214)		$ (214)