Investment in Securities (Unrealized Loss Positions Securities) (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	$ 153,786	$ 471,379
Unrealized Losses, Less than 12 Months	(4,358)	(5,146)
Fair Value, Greater than 12 Months	4,627	162,938
Unrealized Losses, Greater than 12 Months	(1,523)	(3,861)
Fair Value, Total	158,413	634,317
Unrealized Loss, Total	(5,881)	(9,007)
Agency RMBS
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	126,926	328,968
Unrealized Losses, Less than 12 Months	(981)	(1,503)
Fair Value, Greater than 12 Months	832	125,095
Unrealized Losses, Greater than 12 Months	(44)	(1,999)
Fair Value, Total	127,758	454,063
Unrealized Loss, Total	(1,025)	(3,502)
Non-Agency RMBS
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	20,474	88,495
Unrealized Losses, Less than 12 Months	(2,616)	(880)
Fair Value, Greater than 12 Months	2,599	27,218
Unrealized Losses, Greater than 12 Months	(348)	(699)
Fair Value, Total	23,073	115,713
Unrealized Loss, Total	(2,964)	(1,579)
CLOs
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	5,279	37,354
Unrealized Losses, Less than 12 Months	(753)	(1,911)
Fair Value, Greater than 12 Months	1,196	9,245
Unrealized Losses, Greater than 12 Months	(1,131)	(1,103)
Fair Value, Total	6,475	46,599
Unrealized Loss, Total	(1,884)	(3,014)
Other Securities
Securities, Available-for-sale [Line Items]
Fair Value, Less than 12 Months	1,107	16,562
Unrealized Losses, Less than 12 Months	(8)	(852)
Fair Value, Greater than 12 Months	0	1,380
Unrealized Losses, Greater than 12 Months	0	(60)
Fair Value, Total	1,107	17,942
Unrealized Loss, Total	$ (8)	$ (912)