Repurchase Agreements and Other Advances (Details) - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2017	Dec. 31, 2016
Assets Sold under Agreements to Repurchase [Line Items]
Unamortized debt issuance expense	$ 441	$ 210
Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Weighted average remaining term-to-interest rate reset of borrowings under repurchase agreements, days	17 days	19 days
Effective repricing period, months	12 months	12 months
Balance of repurchase agreements	$ 8,137,543	$ 8,472,478
Unamortized debt issuance expense	441	210
Agency MBS | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	3,093,737	3,095,020
Fair value of securities pledged as collateral under repurchase agreements	$ 3,330,251	$ 3,280,689
Weighted average haircut (percent)	4.56%	4.67%
Legacy Non-Agency MBS | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 1,573,545	$ 1,690,937
Fair value of securities pledged as collateral under repurchase agreements	$ 2,070,626	$ 2,317,708
Weighted average haircut (percent)	22.44%	24.01%
Non-Agency MBS acquired from consolidated VIEs		$ 172,400
3 Year Step-up securities | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 1,875,157	2,035,531
Fair value of securities pledged as collateral under repurchase agreements	$ 2,376,119	$ 2,574,691
Weighted average haircut (percent)	21.52%	21.70%
U.S. Treasury securities | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 366,475	$ 504,572
Fair value of securities pledged as collateral under repurchase agreements	$ 371,333	$ 510,767
Weighted average haircut (percent)	1.73%	1.60%
CRT securities | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 345,482	$ 271,205
Fair value of securities pledged as collateral under repurchase agreements	$ 446,231	$ 357,488
Weighted average haircut (percent)	22.36%	23.22%
Residential whole loans | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 836,203	$ 832,060
Fair value of securities pledged as collateral under repurchase agreements	$ 1,178,562	$ 1,175,088
Weighted average haircut (percent)	25.16%	25.03%
Other investments | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 46,944	$ 43,333
Fair value of securities pledged as collateral under repurchase agreements	$ 93,002	$ 85,800
Weighted average haircut (percent)	50.00%	50.00%