Fair Value of Financial Instruments - Carrying Value and Estimated Fair Value of the Company's Financial Assets (Details) - USD ($) $ in Thousands		Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Financial Assets:
Cash and cash equivalents		$ 83,554	$ 61,959	$ 75,598	$ 31,798
Investment securities available for sale	[1]	818,976	765,454
Derivative assets		150,296	228,775
Mezzanine loan and preferred equity investments		156,092	83,995
Financial Liabilities:
Outstanding Borrowings		773,142	577,413
Derivative liabilities		498	1,500
Subordinated debentures		45,000	45,000
Distressed residential mortgage loans, net, held in securitization trusts		195,347	114,214
Financing arrangements, portfolio investments
Financial Liabilities:
Outstanding Borrowings		773,142	577,413
Residential collateralized debt obligations
Financial Liabilities:
Collateralized debt obligations		91,663	116,710
Multi-family collateralized debt obligations, at fair value
Financial Liabilities:
Collateralized debt obligations		6,624,896	6,818,901
Distressed residential mortgage loans held in securitization trusts (net)
Financial Liabilities:
Distressed residential mortgage loans, net, held in securitization trusts		195,300	114,200
Distressed residential mortgage loans
Financial Assets:
Residential mortgage loans held in securitization		307,700	444,800
Financial Liabilities:
Distressed residential mortgage loans, net, held in securitization trusts		503,100	559,000
Level 1 or 2
Financial Assets:
Derivative assets		150,296	228,775
Financial Liabilities:
Derivative liabilities		498	1,500
Level 1
Financial Assets:
Cash and cash equivalents		83,554	61,959
Cash and cash equivalents, estimated fair value		83,554	61,959
Financial Liabilities:
Payables for securities purchased		148,015	227,969
Payables for securities purchased, fair value		148,015	227,969
Level 2 | Financing arrangements, portfolio investments
Financial Liabilities:
Outstanding Borrowings		773,142	577,413
Level 2 | Distressed residential mortgage loans
Financial Liabilities:
Outstanding Borrowings		192,419	212,155
Level 3
Financial Assets:
Mortgage loans held for sale (net)	[2]	7,847	5,471
Mortgage loans held for sale, estimated fair value	[2]	7,959	5,557
Mortgage loans held for investment	[2]	19,529	2,706
Mortgage loans held for investment, estimated fair value	[2]	19,641	2,846
Mezzanine loan and preferred equity investments	[3]	100,150	44,151
Mezzanine loan and preferred equity investments, estimated fair value	[3]	101,408	44,540
Investment in unconsolidated entities	[4]	79,259	87,662
Investments in unconsolidated entities, estimated fair value	[4]	79,390	87,558
Financial Liabilities:
Collateralized debt obligations		158,867	116,541
Collateralized debt obligations, fair value		163,884	123,776
Subordinated debentures		45,000	45,000
Subordinated debentures, fair value		43,132	42,731
Level 3 | Residential collateralized debt obligations
Financial Liabilities:
Collateralized debt obligations		91,663	116,710
Collateralized debt obligations, fair value		85,568	105,606
Level 3 | Multi-family collateralized debt obligations, at fair value
Financial Liabilities:
Collateralized debt obligations		6,624,896	6,818,901
Collateralized debt obligations, fair value		6,624,896	6,818,901
Level 3 | Residential mortgage loans held in securitization trusts
Financial Assets:
Residential mortgage loans held in securitization		95,144	119,921
Residential mortgage loans held in securitization, estimated fair value		88,718	109,120
Level 3 | Distressed residential mortgage loans
Financial Assets:
Residential mortgage loans held in securitization	[5]	503,094	558,989
Residential mortgage loans held in securitization, estimated fair value	[5]	504,915	564,310
Level 3 | Multi-family loans held in securitization trusts, at fair value
Financial Assets:
Residential mortgage loans held in securitization		6,939,844	7,105,336
Residential mortgage loans held in securitization, estimated fair value		6,939,844	7,105,336
Equity Method Investments
Financial Assets:
Investment in unconsolidated entities		18,927	20,091
Investments in unconsolidated entities, estimated fair value		60,332	67,571
Fair Value, Measurements, Recurring
Financial Assets:
Investments in unconsolidated entities, estimated fair value		60,332	67,571
Fair Value, Measurements, Recurring | Level 1
Financial Assets:
Investments in unconsolidated entities, estimated fair value		0	0
Fair Value, Measurements, Recurring | Level 2
Financial Assets:
Investments in unconsolidated entities, estimated fair value		0	0
Fair Value, Measurements, Recurring | Level 3
Financial Assets:
Investments in unconsolidated entities, estimated fair value		60,332	67,571
U.S. Treasury securities
Financial Assets:
Investment securities available for sale	[6]	$ 2,887	$ 10,037

[1] Includes $43.9 million and $40.7 million of investment securities for sale held in securitization trusts as of December 31, 2016 and December 31, 2015, respectively.

[2] Included in receivables and other assets in the accompanying consolidated balance sheets.

[3] Includes mezzanine loan and preferred equity investments accounted for as loans (see Note 8).

[4] Includes investments in unconsolidated entities accounted for under the fair value option with a carrying value of $60.3 million and $67.6 million at December 31, 2016 and December 31, 2015, respectively.

[5] Includes distressed residential mortgage loans held in securitization trusts with a carrying value amounting to approximately $195.3 million and $114.2 million at December 31, 2016 and December 31, 2015, respectively and distressed residential mortgage loans with a carrying value amounting to approximately $307.7 million and $444.8 million at December 31, 2016 and December 31, 2015, respectively.

[6] Included in investment securities available for sale are Agency IOs, Agency RMBS and U.S. Treasury securities managed by Midway that are measured at fair value through earnings.