Stockholders' Equity (Details) $ / shares in Units, $ in Thousands																										12 Months Ended			172 Months Ended
	Dec. 12, 2018 $ / shares	Nov. 26, 2018 $ / shares	Sep. 13, 2018 $ / shares	Aug. 20, 2018 $ / shares	Jun. 07, 2018 $ / shares	May 17, 2018 $ / shares	Mar. 07, 2018 $ / shares	Feb. 20, 2018 $ / shares	Dec. 17, 2017 $ / shares	Nov. 17, 2017 $ / shares	Sep. 14, 2017 $ / shares	Aug. 10, 2017 $ / shares	Jun. 12, 2017 $ / shares	May 16, 2017 $ / shares	Mar. 08, 2017 $ / shares	Feb. 17, 2017 $ / shares	Dec. 14, 2016 $ / shares	Nov. 22, 2016 $ / shares	Sep. 15, 2016 $ / shares	Aug. 12, 2016 $ / shares	Jun. 14, 2016 $ / shares	May 18, 2016 $ / shares	Mar. 11, 2016 $ / shares	Feb. 12, 2016 $ / shares	Apr. 15, 2013 $ / shares shares	Dec. 31, 2018 USD ($) quarter director $ / shares shares	Dec. 31, 2017 USD ($) $ / shares shares	Dec. 31, 2016 USD ($) $ / shares shares	Dec. 31, 2017 USD ($) $ / shares shares	Dec. 31, 2013 shares	Aug. 08, 2013 shares	Aug. 31, 2005 shares
Dividends on Preferred Stock
Preferred stock, shares issued (in shares)																										8,000,000	8,000,000		8,000,000
Preferred stock, dividend rate																									7.50%	7.50%	7.50%
Preferred stock, par value (in dollars per share) | $ / shares																										$ 0.01	$ 0.01		$ 0.01
Maximum quarters without dividends to get voting rights, in quarters | quarter																										6
Minimum percentage of preferred shareholders required for approval																										66.67%
Dividends on Common Stock
Common stock, cash dividends declared (in dollars per share) | $ / shares	$ 0.20		$ 0.20		$ 0.20		$ 0.20		$ 0.20		$ 0.20		$ 0.20		$ 0.20		$ 0.20		$ 0.20		$ 0.20		$ 0.20
Dividends payable | $																										$ 90,198	$ 79,771	$ 74,657	$ 79,771
Residential whole loans, at fair value | $																										$ 1,665,978	$ 1,325,115		$ 1,325,115
Common stock dividends, capital gains | $ / shares																										$ 0.1290	$ 0.0831	$ 0.0361	$ 0.0831
Common stock, shares authorized (in shares)																										886,950,000	886,950,000		886,950,000
Number of shares authorized to be repurchased under the Repurchase Program (in shares)																														10,000,000.0		4,000,000.0
Number of shares repurchased through the Company's publicly announced stock repurchase program (in shares)																										0
Number of remaining shares authorized to be repurchased under the Repurchase Program (in shares)																										6,616,355
DRSPP
Dividends on Common Stock
Common stock, shares authorized (in shares)																															15,000,000
Shares of common stock authorized and available for issuance																										11,800,000.0
Shares issued (in shares)																										379,903	2,293,192	653,793	34,055,880
Net proceeds | $																										$ 2,800	$ 18,500	$ 1,200	$ 284,200
Series B Preferred Stock
Dividends on Preferred Stock
Preferred stock, cash dividends declared (in dollars per share) | $ / shares		$ 0.46875		$ 0.46875		$ 0.46875		$ 0.46875		$ 0.46875		$ 0.46875		$ 0.46875		$ 0.46875		$ 0.46875		$ 0.46875		$ 0.46875		$ 0.46875
Preferred Stock | Series B Preferred Stock
Dividends on Preferred Stock
Preferred stock, shares issued (in shares)																									8,000,000.0
Preferred stock, dividend rate																									7.50%	7.50%	7.50%	7.50%
Preferred stock, par value (in dollars per share) | $ / shares																									$ 0.01
Preferred stock, liquidation preference (in dollars per share) | $ / shares																									25.00	$ 25.00	$ 25	$ 25.00	$ 25
Preferred stock, redemption price (in dollars per share) | $ / shares																									$ 25.00
Number of additional directors that can be elected by preferred stock holders | director																										2