Stockholders' Equity - Components of Accumulated Other Comprehensive Income (Details) - USD ($) $ in Thousands	3 Months Ended						9 Months Ended
	Sep. 30, 2021	Jun. 30, 2021	Mar. 31, 2021	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Sep. 30, 2021	Sep. 30, 2020
Accumulated Other Comprehensive Income (Loss) [Line Items]
Unrealized gain (loss) on mortgage-backed and credit risk transfer securities, net	$ (473)			$ 22,812			$ 1,663	$ (217,064)
Reclassification of unrealized (gain) loss on sale of mortgage-backed and credit risk transfer securities to gain (loss) on investments, net	0			(54,615)			0	17,124
Reclassification of amortization of net deferred (gain) loss on de-designated interest rate swaps to repurchase agreements interest expense	(5,601)			(3,243)			(16,398)	(17,813)
Currency translation adjustments on investment in unconsolidated venture	187			397			164	489
Total other comprehensive income (loss)	(5,887)			(34,649)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	1,373,379	$ 1,481,932	$ 1,367,158	1,157,792	$ 1,410,381	$ 2,931,899	1,367,158	2,931,899
Total other comprehensive income (loss)	(5,887)	(4,906)	(3,778)	(34,649)	(23,380)	(159,235)	(14,571)	(217,264)
Ending balance	1,456,371	1,373,379	1,481,932	1,211,140	1,157,792	1,410,381	1,456,371	1,211,140
Equity method investments
Accumulated Other Comprehensive Income (Loss) [Line Items]
Currency translation adjustments on investment in unconsolidated venture	187			397			164	489
Total other comprehensive income (loss)	187			397
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	476		499	(553)		(645)	499	(645)
Total other comprehensive income (loss)	187			397			164	489
Ending balance	663	476		(156)	(553)		663	(156)
Available-for-sale securities
Accumulated Other Comprehensive Income (Loss) [Line Items]
Unrealized gain (loss) on mortgage-backed and credit risk transfer securities, net	(473)			22,812			1,663	(217,064)
Reclassification of unrealized (gain) loss on sale of mortgage-backed and credit risk transfer securities to gain (loss) on investments, net				(54,615)				17,124
Currency translation adjustments on investment in unconsolidated venture							0
Total other comprehensive income (loss)	(473)			(31,803)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	8,129		5,993	45,564		213,701	5,993	213,701
Total other comprehensive income (loss)	(473)			(31,803)			1,663	(199,940)
Ending balance	7,656	8,129		13,761	45,564		7,656	13,761
Derivative and hedging attributable to Parent
Accumulated Other Comprehensive Income (Loss) [Line Items]
Reclassification of amortization of net deferred (gain) loss on de-designated interest rate swaps to repurchase agreements interest expense	(5,601)			(3,243)			(16,398)	(17,813)
Total other comprehensive income (loss)	(5,601)			(3,243)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	41,316		52,113	61,337		75,907	52,113	75,907
Total other comprehensive income (loss)	(5,601)			(3,243)			(16,398)	(17,813)
Ending balance	35,715	41,316		58,094	61,337		35,715	58,094
Accumulated other comprehensive income
Accumulated Other Comprehensive Income (Loss) [Line Items]
Total other comprehensive income (loss)				(34,649)
AOCI Including Portion Attributable to Noncontrolling Interest, Net of Tax [Roll Forward]
Beginning balance	49,921	54,827	58,605	106,348	129,728	288,963	58,605	288,963
Total other comprehensive income (loss)	(5,887)	(4,906)	(3,778)		(23,380)	(159,235)
Ending balance	$ 44,034	$ 49,921	$ 54,827	$ 71,699	$ 106,348	$ 129,728	$ 44,034	$ 71,699