Investment in Securities (Summary of Investment Income) (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2019	Dec. 31, 2018
Securities, Available-for-sale [Line Items]
Net Amortization									$ (48,132)	$ (45,895)
Interest Income	$ 45,353	$ 39,985	$ 38,547	$ 36,016	$ 35,694	$ 35,300	$ 31,941	$ 28,092	159,901	$ 131,027	[1]
Agency RMBS
Securities, Available-for-sale [Line Items]
Coupon Interest									62,103
Net Amortization									(24,731)
Interest Income									37,372
Non-Agency RMBS and CMBS
Securities, Available-for-sale [Line Items]
Coupon Interest									13,855
Net Amortization									2,782
Interest Income									16,637
CLOs
Securities, Available-for-sale [Line Items]
Coupon Interest									15,857
Net Amortization									(1,599)
Interest Income									14,258
Other Securities
Securities, Available-for-sale [Line Items]
Coupon Interest									7,157
Net Amortization									(2,468)
Interest Income									4,689
Securities Investment [Member]
Securities, Available-for-sale [Line Items]
Coupon Interest									98,972
Net Amortization									(26,016)
Interest Income									$ 72,956

[1] Includes interest income and interest expense of a consolidated securitization trust of $6.0 million and $3.6 million, respectively, for the year ended December 31, 2018. See Note 6 for further details on the Company's consolidated securitization trust.