Fair Value of Financial Instruments - Level 3 Assets (Details) - Recurring basis - Level 3 - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2017	Mar. 31, 2016
Residential whole loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	$ 814,682	$ 623,276
Purchases and capitalized advances	5,333	53,590
Changes in fair value recorded in Net gain on residential whole loans held at fair value	2,948	6,226
Collection of principal, net of liquidation gains/losses	(20,043)	(14,602)
Repurchases	(306)	0
Transfers	(27,462)	(21,130)
Balance at end of period	775,152	647,360
Other investments
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	0	0
Purchases and capitalized advances	150,000	0
Collection of principal, net of liquidation gains/losses	(8,648)	0
Transfers	140,980	0
Balance at end of period	$ 282,332	$ 0