Other Assets and Liabilities - Accrued Expenses and Other Liabilities (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Jun. 30, 2016	Dec. 31, 2015
Deferred Costs, Capitalized, Prepaid, and Other Assets Disclosure [Abstract]
Guarantee obligations	$ 23,011		$ 22,704
Margin payable	13,313		6,415
Accrued compensation	12,674		17,527
Residential loan and MSR repurchase reserve	6,617		6,403
Accrued operating expenses	5,958		1,845
Restructuring liabilities	3,667	$ 4,159	0
Legal reserve	2,000		2,000
Current accounts payable	1,292		4,764
Other	1,176		8,239
Total Other Liabilities	$ 69,708		$ 69,897