Investment in Securities (Unrealized Loss Positions Securities) (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	$ 79,670	$ 471,379
Unrealized Losses, Less than 12 Months	(13,643)	(5,146)
Fair Value, Greater than 12 Months	20,396	162,938
Unrealized Losses, Greater than 12 Months	(894)	(3,861)
Fair Value, Total	100,066	634,317
Unrealized Loss, Total	(14,537)	(9,007)
Agency RMBS
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	64,974	328,968
Unrealized Losses, Less than 12 Months	(520)	(1,503)
Fair Value, Greater than 12 Months	1,119	125,095
Unrealized Losses, Greater than 12 Months	(38)	(1,999)
Fair Value, Total	66,093	454,063
Unrealized Loss, Total	(558)	(3,502)
Non-Agency RMBS
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	11,381	88,495
Unrealized Losses, Less than 12 Months	(2,119)	(880)
Fair Value, Greater than 12 Months	19,277	27,218
Unrealized Losses, Greater than 12 Months	(856)	(699)
Fair Value, Total	30,658	115,713
Unrealized Loss, Total	(2,975)	(1,579)
CLOs
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	2,633	37,354
Unrealized Losses, Less than 12 Months	(10,609)	(1,911)
Fair Value, Greater than 12 Months	0	9,245
Unrealized Losses, Greater than 12 Months	0	(1,103)
Fair Value, Total	2,633	46,599
Unrealized Loss, Total	(10,609)	(3,014)
Other Securities
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	682	16,562
Unrealized Losses, Less than 12 Months	(395)	(852)
Fair Value, Greater than 12 Months	0	1,380
Unrealized Losses, Greater than 12 Months	0	(60)
Fair Value, Total	682	17,942
Unrealized Loss, Total	$ (395)	$ (912)