Summary of Quarterly Results of Operations (Unaudited) (Details) - USD ($) $ / shares in Units, $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Quarterly Financial Information Disclosure [Abstract]
Interest income	$ 153,118	$ 142,721	$ 144,935	$ 140,952	$ 132,744	$ 117,432	$ 101,747	$ 103,752	$ 581,726	$ 455,675	$ 433,448
Interest expense	(82,463)	(85,823)	(85,044)	(79,026)	(70,944)	(58,878)	(51,810)	(50,554)	(332,356)	(232,186)	(197,141)
Net Interest Income	70,655	56,898	59,891	61,926	61,800	58,554	49,937	53,198	249,370	223,489	236,307
Net gain on residential whole loans measured at fair value through earnings	41,415	40,175	51,473	25,267	31,736	34,942	32,443	38,498	158,330	137,619	90,045
Net realized gain on sales of residential mortgage securities	11,975	17,708	7,710	24,609	28,646	16,415	7,429	8,817
Other income	2,007	4,546	(2,321)	1,293	(39,432)	(2,998)	1,134	345
Operating and other expense	(25,431)	(23,728)	(23,713)	(24,238)	(21,871)	(19,781)	(20,548)	(17,463)	(97,110)	(79,663)	(71,901)
Net Income	100,621	95,599	93,040	88,857	60,879	87,132	70,395	83,395	378,117	301,801	322,393
Preferred stock dividends	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(3,750)	(15,000)	(15,000)	(15,000)
Net income available to common stock and participating securities	$ 96,871	$ 91,849	$ 89,290	$ 85,107	$ 57,129	$ 83,382	$ 66,645	$ 79,645	$ 362,030	$ 285,858	$ 306,502
Earnings per Common Share - Basic and Diluted (in dollars per share)	$ 0.21	$ 0.20	$ 0.20	$ 0.19	$ 0.13	$ 0.19	$ 0.17	$ 0.20