Other Assets - Interest Receivable (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Dec. 31, 2015
Interest receivable by investment category
Total interest receivable	$ 27,842	$ 29,002
Total MBS interest receivable
Interest receivable by investment category
Total interest receivable	24,131	26,629
Fannie Mae
Interest receivable by investment category
Total interest receivable	7,799	8,999
Freddie Mac
Interest receivable by investment category
Total interest receivable	1,916	2,177
Ginnie Mae
Interest receivable by investment category
Total interest receivable	15	15
Non-Agency MBS
Interest receivable by investment category
Total interest receivable	14,401	15,438
Residential whole loans
Interest receivable by investment category
Total interest receivable	3,435	2,259
CRT securities
Interest receivable by investment category
Total interest receivable	213	92
Money market and other investments
Interest receivable by investment category
Total interest receivable	$ 63	$ 22