Investment in Securities (Summary of Investment Income) (Details) - USD ($) $ in Thousands	3 Months Ended												12 Months Ended
	Dec. 31, 2020	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Securities, Available-for-sale [Line Items]
Net Amortization													$ (41,681)	$ (48,132)	$ (45,895)
Interest Income	$ 39,067	$ 43,075	$ 39,281	$ 52,108	$ 45,353	$ 39,985	$ 38,547	$ 36,016	$ 35,694	$ 35,300	$ 31,941	$ 28,092	173,531	159,901	$ 131,027	[1]
Agency RMBS
Securities, Available-for-sale [Line Items]
Coupon Interest													59,987	62,103
Net Amortization													(31,975)	(24,731)
Interest Income													28,012	37,372
Non-Agency RMBS and CMBS
Securities, Available-for-sale [Line Items]
Coupon Interest													15,257	13,855
Net Amortization													5,775	2,782
Interest Income													21,032	16,637
CLOs
Securities, Available-for-sale [Line Items]
Coupon Interest													14,103	15,857
Net Amortization													3,529	(1,599)
Interest Income													17,632	14,258
Other Securities
Securities, Available-for-sale [Line Items]
Coupon Interest													11,545	7,157
Net Amortization													(4,986)	(2,468)
Interest Income													6,559	4,689
Securities Investment [Member]
Securities, Available-for-sale [Line Items]
Coupon Interest													100,892	98,972
Net Amortization													(27,657)	(26,016)
Interest Income													$ 73,235	$ 72,956

[1] Includes interest income and interest expense of a consolidated securitization trust of $6.0 million and $3.6 million, respectively, for the year ended December 31, 2018. See Note 6 for further details on the Company's consolidated securitization trust.