Investment in Securities (Unrealized Loss Positions Securities) (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	$ 331,085	$ 471,379
Unrealized Losses, Less than 12 Months	(86,418)	(5,146)
Fair Value, Greater than 12 Months	19,392	162,938
Unrealized Losses, Greater than 12 Months	(5,347)	(3,861)
Fair Value, Total	350,477	634,317
Unrealized Loss, Total	(91,765)	(9,007)
Agency RMBS
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	3,912	328,968
Unrealized Losses, Less than 12 Months	(342)	(1,503)
Fair Value, Greater than 12 Months	852	125,095
Unrealized Losses, Greater than 12 Months	(78)	(1,999)
Fair Value, Total	4,764	454,063
Unrealized Loss, Total	(420)	(3,502)
Non-Agency RMBS
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	119,313	88,495
Unrealized Losses, Less than 12 Months	(30,678)	(880)
Fair Value, Greater than 12 Months	13,070	27,218
Unrealized Losses, Greater than 12 Months	(1,142)	(699)
Fair Value, Total	132,383	115,713
Unrealized Loss, Total	(31,820)	(1,579)
CLOs
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	159,024	37,354
Unrealized Losses, Less than 12 Months	(55,065)	(1,911)
Fair Value, Greater than 12 Months	5,470	9,245
Unrealized Losses, Greater than 12 Months	(4,127)	(1,103)
Fair Value, Total	164,494	46,599
Unrealized Loss, Total	(59,192)	(3,014)
Other Securities
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	48,836	16,562
Unrealized Losses, Less than 12 Months	(333)	(852)
Fair Value, Greater than 12 Months	0	1,380
Unrealized Losses, Greater than 12 Months	0	(60)
Fair Value, Total	48,836	17,942
Unrealized Loss, Total	$ (333)	$ (912)