Repurchase Agreements (Details) - USD ($) $ in Thousands	Sep. 30, 2017	Dec. 31, 2016
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Balance	$ 2,519,230	$ 2,898,952
Weighted Average Rate	1.51%	1.03%
Fair Value of Collateral Pledged	$ 2,718,003	$ 3,149,839
Securitization financing bond
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Balance	$ 3,366	$ 4,803
Weighted Average Rate	2.58%	2.00%
Fair Value of Collateral Pledged	$ 3,723	$ 5,278
CMBS | Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Balance	$ 1,188,230	$ 1,005,726
Weighted Average Rate	1.31%	0.82%
Fair Value of Collateral Pledged	$ 1,250,575	$ 1,095,002
CMBS | Non-Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Balance	$ 15,625	$ 66,881
Weighted Average Rate	2.14%	1.63%
Fair Value of Collateral Pledged	$ 18,365	$ 77,840
CMBS IO | Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Balance	$ 336,187	$ 346,892
Weighted Average Rate	2.06%	1.57%
Fair Value of Collateral Pledged	$ 389,637	$ 407,481
CMBS IO | Non-Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Balance	$ 279,981	$ 291,199
Weighted Average Rate	2.15%	1.67%
Fair Value of Collateral Pledged	$ 327,976	$ 341,139
RMBS | Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Balance	$ 695,841	$ 1,157,302
Weighted Average Rate	1.31%	0.82%
Fair Value of Collateral Pledged	$ 727,727	$ 1,191,147
RMBS | Non-Agency MBS
Assets and Associated Liabilities of Transfers Accounted for as Secured Borrowings [Line Items]
Balance	$ 0	$ 26,149
Weighted Average Rate	0.00%	1.98%
Fair Value of Collateral Pledged	$ 0	$ 31,952