Valuation (Significant Unobservable Inputs Rollforward) (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2019	Jun. 30, 2018	Jun. 30, 2019	Jun. 30, 2018
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Transfers into Level 3				$ 3,100
Assets, Transfers out of Level 3				(51,600)
Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gains (losses), liabilities	$ (17)		$ 40
Securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	5,700		6,100
Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	3,800		6,000
Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	2,000		(200)
Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Change in unrealized gain (loss), assets	(56)		(300)
Level 3
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,233,319	$ 758,723	1,094,815	661,588
Assets, Accreted Discounts/Amortized Premiums	(8,631)	(9,827)	(19,524)	(14,251)
Assets, Realized Gain (Loss)	(258)	3,216	(2,591)	7,944
Assets, Change In Net Unrealized Gain/(Loss)	8,221	1,848	12,231	11,688
Assets, Purchases	200,759	223,767	496,611	386,854
Assets, Sales	(110,689)	(112,690)	274,911	(202,131)
Assets, Transfers into Level 3	15,320	3,073	24,384	9,301
Assets, Transfers out of Level 3	13,450	(51,604)	6,424	(44,487)
Assets, Ending Balance	1,324,591	816,506	1,324,591	816,506
Liabilities, Begining Balance	282,124	(113,775)	(297,948)	(125,105)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	0	0	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(17)	414	40	1,198
Liabilities Purchases/Payments	25,861	12,261	41,628	22,807
Liabilities, Sales/Issuance	(219,536)	0	(219,536)	0
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(475,816)	(101,100)	(475,816)	(101,100)
Level 3 | Other secured borrowings, at fair value
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Liabilities, Begining Balance	282,124	(113,775)	(297,948)	(125,105)
Liabilities, Accreted Discounts/Amortized Premiums	0	0	0	0
Liabilities, Realized Gain/(Loss)	0	0	0	0
Liabilities, Change In Net Unrealized Gain/(Loss)	(17)	414	40	1,198
Liabilities, Payments	25,861	12,261	41,628	22,807
Liabilities, Issuances	(219,536)	0	(219,536)	0
Liabilities, Transfers into Level 3	0	0	0	0
Liabilities, Transfers out of Level 3	0	0	0	0
Liabilities, Ending Balance	(475,816)	(101,100)	(475,816)	(101,100)
Change in unrealized gains (losses), liabilities		400		1,200
Level 3 | Securities | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	6,389		7,293
Assets, Accreted Discounts/Amortized Premiums	(676)		(1,449)
Assets, Realized Gain (Loss)	(450)		(1,075)
Assets, Change In Net Unrealized Gain/(Loss)	349		642
Assets, Purchases	1,594		1,600
Assets, Sales	(236)		236
Assets, Transfers into Level 3	4,889		5,596
Assets, Transfers out of Level 3	825		1,337
Assets, Ending Balance	11,034		11,034
Level 3 | Securities | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	94,670		91,291
Assets, Accreted Discounts/Amortized Premiums	108		195
Assets, Realized Gain (Loss)	(384)		(650)
Assets, Change In Net Unrealized Gain/(Loss)	2,909		2,358
Assets, Purchases	2,050		17,595
Assets, Sales	(3,294)		20,220
Assets, Transfers into Level 3	6,366		11,308
Assets, Transfers out of Level 3	5,635		5,087
Assets, Ending Balance	96,790		96,790
Level 3 | Securities | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	5,137		803
Assets, Accreted Discounts/Amortized Premiums	25		(23)
Assets, Realized Gain (Loss)	50		0
Assets, Change In Net Unrealized Gain/(Loss)	363		(7)
Assets, Purchases	0		0
Assets, Sales	(374)		0
Assets, Transfers into Level 3	1,077		5,505
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	6,278		6,278
Level 3 | Securities | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	21,438		14,915
Assets, Accreted Discounts/Amortized Premiums	165		(483)
Assets, Realized Gain (Loss)	(1,151)		(1,676)
Assets, Change In Net Unrealized Gain/(Loss)	772		1,660
Assets, Purchases	0		831
Assets, Sales	0		0
Assets, Transfers into Level 3	2,988		1,975
Assets, Transfers out of Level 3	6,990		0
Assets, Ending Balance	17,222		17,222
Level 3 | Securities | Asset-backed securities, backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	24,108		22,800
Assets, Accreted Discounts/Amortized Premiums	(360)		(969)
Assets, Realized Gain (Loss)	(152)		(665)
Assets, Change In Net Unrealized Gain/(Loss)	95		857
Assets, Purchases	4,610		9,551
Assets, Sales	(3,282)		6,555
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	25,019		25,019
Level 3 | Securities | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	5,737		6,318
Assets, Accreted Discounts/Amortized Premiums	6		22
Assets, Realized Gain (Loss)	(345)		(345)
Assets, Change In Net Unrealized Gain/(Loss)	(1)		(78)
Assets, Purchases	2,449		2,832
Assets, Sales	(3,765)		4,668
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	4,081		4,081
Level 3 | Securities | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,469		1,534
Assets, Accreted Discounts/Amortized Premiums	0		0
Assets, Realized Gain (Loss)	(142)		(142)
Assets, Change In Net Unrealized Gain/(Loss)	464		399
Assets, Purchases	0		0
Assets, Sales	0		0
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	1,791		1,791
Level 3 | Investments
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		755,654		658,448
Assets, Accreted Discounts/Amortized Premiums		(9,827)		(14,251)
Assets, Realized Gain (Loss)		3,055		7,697
Assets, Change In Net Unrealized Gain/(Loss)		2,326		12,237
Assets, Purchases		223,745		386,809
Assets, Sales		(112,507)		(201,839)
Assets, Transfers into Level 3		3,073		9,301
Assets, Transfers out of Level 3		(51,604)		(44,487)
Assets, Ending Balance		813,915		813,915
Change in unrealized gain (loss), assets		1,600		10,500
Level 3 | Investments | Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		6,128		6,173
Assets, Accreted Discounts/Amortized Premiums		(547)		(1,172)
Assets, Realized Gain (Loss)		18		57
Assets, Change In Net Unrealized Gain/(Loss)		(41)		229
Assets, Purchases		1,533		2,635
Assets, Sales		(569)		(957)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		(633)		(1,076)
Assets, Ending Balance		5,889		5,889
Level 3 | Investments | Non-Agency RMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		111,862		101,297
Assets, Accreted Discounts/Amortized Premiums		(1,889)		156
Assets, Realized Gain (Loss)		211		2,719
Assets, Change In Net Unrealized Gain/(Loss)		(1,845)		(986)
Assets, Purchases		13,596		24,623
Assets, Sales		(10,999)		(29,439)
Assets, Transfers into Level 3		3,073		9,301
Assets, Transfers out of Level 3		(17,613)		(11,275)
Assets, Ending Balance		96,396		96,396
Level 3 | Investments | CMBS
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		13,709		12,347
Assets, Accreted Discounts/Amortized Premiums		28		(163)
Assets, Realized Gain (Loss)		(104)		1,618
Assets, Change In Net Unrealized Gain/(Loss)		784		1,043
Assets, Purchases		0		8,046
Assets, Sales		(4,194)		(13,890)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		(1,462)		(240)
Assets, Ending Balance		8,761		8,761
Level 3 | Investments | CLOs
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		27,479		24,911
Assets, Accreted Discounts/Amortized Premiums		(377)		78
Assets, Realized Gain (Loss)		2		3
Assets, Change In Net Unrealized Gain/(Loss)		300		527
Assets, Purchases		11,442		21,537
Assets, Sales		(7,889)		(16,099)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		(24,848)		(24,848)
Assets, Ending Balance		6,109		6,109
Level 3 | Investments | Corporate debt securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		18,000		23,947
Assets, Accreted Discounts/Amortized Premiums		59		(56)
Assets, Realized Gain (Loss)		96		148
Assets, Change In Net Unrealized Gain/(Loss)		(761)		(396)
Assets, Purchases		448		905
Assets, Sales		(1,944)		(8,650)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		(7,048)		(7,048)
Assets, Ending Balance		8,850		8,850
Level 3 | Investments | Secured notes
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		0		0
Assets, Accreted Discounts/Amortized Premiums		92		92
Assets, Realized Gain (Loss)		0		0
Assets, Change In Net Unrealized Gain/(Loss)		(234)		(234)
Assets, Purchases		11,268		11,268
Assets, Sales		0		0
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		11,126		11,126
Level 3 | Investments | Corporate equity securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		50,869		37,465
Assets, Accreted Discounts/Amortized Premiums		0		0
Assets, Realized Gain (Loss)		1,182		1,182
Assets, Change In Net Unrealized Gain/(Loss)		(1)		4,325
Assets, Purchases		0		9,078
Assets, Sales		(7,282)		(7,282)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		44,768		44,768
Level 3 | Investments | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		240,781		182,472
Assets, Accreted Discounts/Amortized Premiums		(464)		(1,179)
Assets, Realized Gain (Loss)		637		583
Assets, Change In Net Unrealized Gain/(Loss)		(267)		(920)
Assets, Purchases		76,785		149,825
Assets, Sales		(24,000)		(37,309)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		293,472		293,472
Level 3 | Investments | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		109,294		108,301
Assets, Accreted Discounts/Amortized Premiums		226		844
Assets, Realized Gain (Loss)		553		884
Assets, Change In Net Unrealized Gain/(Loss)		610		449
Assets, Purchases		20,132		24,119
Assets, Sales		(25,864)		(29,646)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		104,951		104,951
Level 3 | Investments | Consumer loans and asset-backed securities backed by consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		148,422		135,258
Assets, Accreted Discounts/Amortized Premiums		(6,955)		(12,851)
Assets, Realized Gain (Loss)		450		950
Assets, Change In Net Unrealized Gain/(Loss)		3,782		7,585
Assets, Purchases		83,069		125,203
Assets, Sales		(29,514)		(56,891)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		199,254		199,254
Level 3 | Investments | Real estate owned
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		29,110		26,277
Assets, Accreted Discounts/Amortized Premiums		0		0
Assets, Realized Gain (Loss)		10		(447)
Assets, Change In Net Unrealized Gain/(Loss)		(1)		615
Assets, Purchases		5,472		9,570
Assets, Sales		(252)		(1,676)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		34,339		34,339
Level 3 | Loans | Residential mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	583,252		496,829
Assets, Accreted Discounts/Amortized Premiums	(1,347)		(2,274)
Assets, Realized Gain (Loss)	2,089		1,954
Assets, Change In Net Unrealized Gain/(Loss)	2,257		4,158
Assets, Purchases	103,844		261,446
Assets, Sales	(26,215)		98,233
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	663,880		663,880
Level 3 | Loans | Commercial mortgage loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	239,623		195,301
Assets, Accreted Discounts/Amortized Premiums	834		1,139
Assets, Realized Gain (Loss)	1,413		1,414
Assets, Change In Net Unrealized Gain/(Loss)	(2,018)		(2,352)
Assets, Purchases	47,555		96,412
Assets, Sales	(27,373)		31,880
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	260,034		260,034
Level 3 | Loans | Consumer loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	192,115		183,961
Assets, Accreted Discounts/Amortized Premiums	(7,386)		(15,958)
Assets, Realized Gain (Loss)	(1,455)		(3,510)
Assets, Change In Net Unrealized Gain/(Loss)	857		2,699
Assets, Purchases	16,626		70,882
Assets, Sales	(38,148)		75,465
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	162,609		162,609
Level 3 | Loans | Corporate loan [Member]
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	0		0
Assets, Accreted Discounts/Amortized Premiums	0		0
Assets, Realized Gain (Loss)	0		0
Assets, Change In Net Unrealized Gain/(Loss)	0		0
Assets, Purchases	5,000		5,000
Assets, Sales	0		0
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	5,000		5,000
Level 3 | Investment in unconsolidated entities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	58,148		72,298
Assets, Accreted Discounts/Amortized Premiums	0		276
Assets, Realized Gain (Loss)	124		1,684
Assets, Change In Net Unrealized Gain/(Loss)	2,230		2,190
Assets, Purchases	17,026		30,454
Assets, Sales	(7,852)		37,226
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	69,676		69,676
Level 3 | Financial Derivatives - Assets
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance		3,069		3,140
Assets, Accreted Discounts/Amortized Premiums		0		0
Assets, Realized Gain (Loss)		161		247
Assets, Change In Net Unrealized Gain/(Loss)		(478)		(549)
Assets, Purchases		22		45
Assets, Sales		(183)		(292)
Assets, Transfers into Level 3		0		0
Assets, Transfers out of Level 3		0		0
Assets, Ending Balance		2,591		2,591
Change in unrealized gain (loss), assets		(500)		(500)
Level 3 | Financial Derivatives - Assets | Total return swaps
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	0		0
Assets, Accreted Discounts/Amortized Premiums	0		0
Assets, Realized Gain (Loss)	17		17
Assets, Change In Net Unrealized Gain/(Loss)	87		87
Assets, Purchases	0		0
Assets, Sales	(17)		17
Assets, Transfers into Level 3	0		0
Assets, Transfers out of Level 3	0		0
Assets, Ending Balance	87		87
Level 3 | Financial Derivatives - Assets | Credit default swaps on asset-backed securities
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Assets, Beginning Balance	1,233	3,069	1,472	3,140
Assets, Accreted Discounts/Amortized Premiums	0	0	0	0
Assets, Realized Gain (Loss)	128	161	403	247
Assets, Change In Net Unrealized Gain/(Loss)	(143)	(478)	(382)	(549)
Assets, Purchases	5	22	8	45
Assets, Sales	(133)	(183)	411	(292)
Assets, Transfers into Level 3	0	0	0	0
Assets, Transfers out of Level 3	0	0	0	0
Assets, Ending Balance	$ 1,090	$ 2,591	$ 1,090	$ 2,591