Summary of Quarterly Results of Operations (Unaudited) (Details) - USD ($) $ / shares in Units, $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Quarterly Financial Information Disclosure [Abstract]
Interest income	$ 100,901	$ 105,133	$ 110,157	$ 117,257	$ 112,809	$ 112,716	$ 114,507	$ 117,418	$ 433,448	$ 457,450	$ 492,143
Interest expense	(48,495)	(49,275)	(49,022)	(50,349)	(49,868)	(48,167)	(47,720)	(47,600)	(197,141)	(193,355)	(176,948)
Net Interest Income	52,406	55,858	61,135	66,908	62,941	64,549	66,787	69,818	236,307	264,095	315,195
Net impairment losses recognized in earnings	0	0	(618)	(414)	0	(485)	0	0	(1,032)	(485)	(705)
Net gain on residential whole loans held at fair value	41,359	18,679	16,208	13,773	14,876	19,639	15,742	12,348	90,019	62,605	19,575
Net gain on sales of MBS and U.S. Treasury securities	9,047	14,933	5,889	9,708	9,768	7,083	9,241	9,745	39,577	35,837	34,900
Other income	14,579	(4,515)	14,847	4,512	756	7,179	2,047	618
Operating and other expense	(16,798)	(21,150)	(17,526)	(16,427)	(15,704)	(14,954)	(14,867)	(14,459)	(71,901)	(59,984)	(52,429)
Net Income	100,593	63,805	79,935	78,060	72,637	83,011	78,950	78,070	322,393	312,668	313,226
Preferred stock dividends	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(15,000)	(15,000)	(15,000)
Net Income Available to Common Stock and Participating Securities	$ 96,843	$ 60,055	$ 76,185	$ 74,310	$ 68,887	$ 79,261	$ 75,200	$ 74,320	$ 307,393	$ 297,668	$ 298,226
Earnings per Common Share - Basic and Diluted	$ 0.24	$ 0.15	$ 0.20	$ 0.20	$ 0.18	$ 0.21	$ 0.20	$ 0.20	$ 0.79	$ 0.80	$ 0.80