Real Estate Securities - Gross Realized Gains and Losses on Sales and Calls of Available for Sale Securities (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015
Schedule of Available-for-sale Securities [Line Items]
Gross realized gains	$ 1,990	$ 4,053	$ 22,395	$ 14,315
Gross realized losses	0	0	(2,293)	0
Total Realized Gains on Sales and Calls of AFS Securities, net	1,990	5,548	21,312	16,170
Calls
Schedule of Available-for-sale Securities [Line Items]
Gross realized gains	0	1,607	1,210	1,967
Gross realized losses	$ 0	$ (112)	$ 0	$ (112)