Investment in Securities (Unrealized Loss Positions Securities) (Details) $ in Thousands	Dec. 31, 2019 USD ($)
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	$ 471,379
Unrealized Losses, Less than 12 Months	(5,146)
Fair Value, Greater than 12 Months	162,938
Unrealized Losses, Greater than 12 Months	(3,861)
Fair Value, Total	634,317
Unrealized Loss, Total	(9,007)
Agency RMBS
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	328,968
Unrealized Losses, Less than 12 Months	(1,503)
Fair Value, Greater than 12 Months	125,095
Unrealized Losses, Greater than 12 Months	(1,999)
Fair Value, Total	454,063
Unrealized Loss, Total	(3,502)
Non-Agency RMBS
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	88,495
Unrealized Losses, Less than 12 Months	(880)
Fair Value, Greater than 12 Months	27,218
Unrealized Losses, Greater than 12 Months	(699)
Fair Value, Total	115,713
Unrealized Loss, Total	(1,579)
CLOs
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	37,354
Unrealized Losses, Less than 12 Months	(1,911)
Fair Value, Greater than 12 Months	9,245
Unrealized Losses, Greater than 12 Months	(1,103)
Fair Value, Total	46,599
Unrealized Loss, Total	(3,014)
Other Securities
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	16,562
Unrealized Losses, Less than 12 Months	(852)
Fair Value, Greater than 12 Months	1,380
Unrealized Losses, Greater than 12 Months	(60)
Fair Value, Total	17,942
Unrealized Loss, Total	$ (912)