Investment in Securities (Unrealized Loss Positions Securities) (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Dec. 31, 2019
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	$ 168,024	$ 471,379
Unrealized Losses, Less than 12 Months	(12,733)	(5,146)
Fair Value, Greater than 12 Months	3,600	162,938
Unrealized Losses, Greater than 12 Months	(196)	(3,861)
Fair Value, Total	171,624	634,317
Unrealized Loss, Total	(12,929)	(9,007)
Agency RMBS
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	138,286	328,968
Unrealized Losses, Less than 12 Months	(673)	(1,503)
Fair Value, Greater than 12 Months	1,881	125,095
Unrealized Losses, Greater than 12 Months	(112)	(1,999)
Fair Value, Total	140,167	454,063
Unrealized Loss, Total	(785)	(3,502)
Non-Agency RMBS
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	18,887	88,495
Unrealized Losses, Less than 12 Months	(1,560)	(880)
Fair Value, Greater than 12 Months	1,665	27,218
Unrealized Losses, Greater than 12 Months	(84)	(699)
Fair Value, Total	20,552	115,713
Unrealized Loss, Total	(1,644)	(1,579)
CLOs
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	10,204	37,354
Unrealized Losses, Less than 12 Months	(10,069)	(1,911)
Fair Value, Greater than 12 Months	54	9,245
Unrealized Losses, Greater than 12 Months	0	(1,103)
Fair Value, Total	10,258	46,599
Unrealized Loss, Total	(10,069)	(3,014)
Other Securities
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	647	16,562
Unrealized Losses, Less than 12 Months	(431)	(852)
Fair Value, Greater than 12 Months	0	1,380
Unrealized Losses, Greater than 12 Months	0	(60)
Fair Value, Total	647	17,942
Unrealized Loss, Total	$ (431)	$ (912)