Repurchase Agreements - Narrative (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2019 USD ($) counterparty	Dec. 31, 2018 USD ($) counterparty	Dec. 31, 2017 USD ($)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted Average Months to Maturity	73 days	62 days
Accrued interest payable	$ 8,800	$ 3,900
Advance rate (as a percent)	85.10%
Average haircut (as a percent)	14.90%
Repurchase agreement, number of counterparties | counterparty	14	11
Cash and cash equivalents	$ 118,763	$ 103,724	$ 95,191
Unencumbered investment securities	$ 535,800
Liquidation proceeds (as a percent)	27.80%
Stockholders' Equity, Total | Lender Concentration Risk
Accounts, Notes, Loans and Financing Receivable [Line Items]
Average haircut (as a percent)		5.00%
Concentration risk, percentage		5.04%
Agency RMBS
Accounts, Notes, Loans and Financing Receivable [Line Items]
Average haircut (as a percent)	5.00%
Agency CMBS
Accounts, Notes, Loans and Financing Receivable [Line Items]
Average haircut (as a percent)	5.00%
Non-Agency RMBS
Accounts, Notes, Loans and Financing Receivable [Line Items]
Average haircut (as a percent)	25.00%
CMBS
Accounts, Notes, Loans and Financing Receivable [Line Items]
Average haircut (as a percent)	19.00%
Loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Weighted average interest rate (as a percent)	2.72%	3.41%
Distressed residential mortgage loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Unamortized deferred costs	$ 800	$ 1,200