Fair Value of Financial Instruments - Valuation for Level 3 Liabilities (Details) - USD ($) $ in Thousands		1 Months Ended	12 Months Ended
		Feb. 28, 2015	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period			$ 6,818,901	$ 8,048,053	$ 7,871,020
Total gains (realized/unrealized)
Included in earnings	[1]		(57,687)	(133,245)	260,872
Purchases/(Sales)	[2]		0	(1,009,942)	0
Paydowns			(136,318)	(85,965)	(83,839)
Balance at the end of period			$ 6,624,896	$ 6,818,901	$ 8,048,053
PO Security
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Proceeds from sale of investments		$ 44,300
Gain on sale of investments		1,500
Multi-family collaterized mortgage backed securities
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Deconsolidation of assets		1,100,000
Multi-family collateralized debt obligations, at fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Deconsolidation of assets		$ 1,000,000

[1] Amounts included in interest expense on multi-family collateralized debt obligations, realized gain (loss) on investment securities and related hedges, net and unrealized gain on multi-family loans and debt held in securitization trusts, net.

[2] Included in receivables and other assets in the accompanying consolidated balance sheets.