Use of Special Purpose Entities (SPE) and Variable Interest Entities (VIE) - Assets and Liabilities of Consolidated VIEs (Details) $ in Thousands		1 Months Ended
		Feb. 28, 2015 USD ($)	Dec. 31, 2015 USD ($) securitization		Dec. 26, 2015 USD ($)	Dec. 31, 2014 USD ($) securitization		Dec. 31, 2012 USD ($)
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Investment securities, available for sale, at fair value held in securitization trusts			$ 40,734	[1]		$ 38,594	[2]
Multi-family loans held in securitization trusts, at fair value			7,105,336			8,365,514
Receivables and other assets			219,130			222,491
Total assets	[3]		9,059,564			10,540,005
Securitized debt			117,528			232,877		$ 38,700
Accrued expenses and other liabilities			59,527			74,692
Total liabilities	[3]		8,179,038			9,722,078
Residential collateralized debt obligations
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations			116,710			145,542
Multi-family collateralized debt obligations, at fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations			6,818,901			8,048,053
Residential mortgage loans held in securitization trusts (net)
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential mortgage loans			119,921			149,614
Distressed residential mortgage loans held in securitization trusts (net)
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential mortgage loans			114,214	[4]	$ 5,500	221,591
PO Security
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Proceeds from sale of investments		$ 44,300
Gain on sale of investments		1,500
Multi-family collateralized debt obligations, at fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Deconsolidation of assets		1,000,000
Multi-Family Collateralized Mortgage Backed Securities
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Deconsolidation of assets		$ 1,100,000
Multi-Family Collateralized Mortgage Backed Securities
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Multi-family loans held in securitization trusts, at fair value			1,224,036	[1]		1,273,633	[2]
Receivables and other assets			4,864	[1]		5,097	[2]
Total assets			1,269,634	[1]		1,317,324	[2]
Securitized debt	[5],[6]		28,019	[1]		27,660	[2]
Accrued expenses and other liabilities			4,436	[1]		4,581	[2]
Total liabilities			1,200,925	[1]		1,253,796	[2]
Multi-Family Collateralized Mortgage Backed Securities | Multi-family collateralized debt obligations, at fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations			$ 1,168,470	[1]		$ 1,221,555	[2]
Multi-Family Collateralized Mortgage Backed Securities | K-Series
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Number of securitizations | securitization			2			2
Number of consolidated securitizations | securitization			1			1
Multi-Family Collateralized Mortgage Backed Securities | Residential mortgage loans held in securitization trusts
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential mortgage loans			$ 0	[1]		$ 0	[2]
Collateralized Recourse Financing
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Multi-family loans held in securitization trusts, at fair value			4,633,061	[7]		4,720,908	[8]
Receivables and other assets			15,281	[7]		15,631	[8]
Total assets			4,648,342	[7]		4,736,539	[8]
Securitized debt	[5],[9]		55,853	[7]		55,853	[8]
Accrued expenses and other liabilities			14,750	[7]		14,639	[8]
Total liabilities			4,534,943	[7]		4,628,557	[8]
Collateralized Recourse Financing | Multi-family collateralized debt obligations, at fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations			$ 4,464,340	[7]		$ 4,558,065	[8]
Collateralized Recourse Financing | K-Series
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Number of securitizations | securitization			3			3
Collateralized Recourse Financing | Residential mortgage loans held in securitization trusts
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential mortgage loans			$ 0	[7]		$ 0	[8]
Residential Distressed Mortgage Loan Securitization
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Receivables and other assets			6,076	[4]		39,084
Total assets			120,290	[4]		260,675
Securitized debt	[5],[10]		33,656	[4]		149,364
Accrued expenses and other liabilities			368	[4]		1,024
Total liabilities			$ 34,024	[4]		150,388
Number of securitizations | securitization			4
Residential Distressed Mortgage Loan Securitization | Residential mortgage loans held in securitization trusts
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential mortgage loans			$ 0	[4]		0
Residential mortgage loans held in securitization trusts
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Receivables and other assets			1,200			1,545
Total assets			121,121			151,159
Accrued expenses and other liabilities			13			14
Total liabilities			116,723			145,556
Non-Financings, Multi-Family CMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Multi-family loans held in securitization trusts, at fair value			1,248,239	[11]		2,370,973	[12]
Receivables and other assets			5,456	[11]		10,408	[12]
Total assets			1,253,695	[11]		2,381,381	[12]
Accrued expenses and other liabilities			5,456	[11]		10,304	[12]
Total liabilities			1,191,547	[11]		2,278,737	[12]
Non-Financings, Multi-Family CMBS | Multi-family collateralized debt obligations, at fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations			$ 1,186,091	[11]		$ 2,268,433	[12]
Non-Financings, Multi-Family CMBS | K-Series
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Number of securitizations considered non-financing VIEs | securitization			1			2
Non-Financings, Multi-Family CMBS | Residential mortgage loans held in securitization trusts
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential mortgage loans			$ 0	[11]		$ 0	[12]
Financing and Non-Financing VIEs
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Investment securities, available for sale, at fair value held in securitization trusts			40,734			38,594
Multi-family loans held in securitization trusts, at fair value			7,105,336			8,365,514
Receivables and other assets			32,877			71,765
Total assets			7,413,082			8,847,078
Securitized debt			117,528			232,877
Accrued expenses and other liabilities			25,023			30,562
Total liabilities			7,078,162			8,457,034
Financing and Non-Financing VIEs | Residential collateralized debt obligations
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations			116,710			145,542
Financing and Non-Financing VIEs | Multi-family collateralized debt obligations, at fair value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Collateralized debt obligations			6,818,901			8,048,053
Financing and Non-Financing VIEs | Distressed residential mortgage loans held in securitization trusts (net)
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential mortgage loans			114,214			221,591
Financing and Non-Financing VIEs | Residential mortgage loans held in securitization trusts
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Residential mortgage loans			$ 119,921			$ 149,614

[1] The Company classified the multi-family CMBS issued by two K-Series securitizations and held by this Financing VIE as available for sale securities as the purpose is not to trade these securities. The Financing VIE consolidated one K-Series securitization that issued certain of the multi-family CMBS owned by the Company, including its assets, liabilities, income and expenses, in its financial statements, as based on a number of factors, the Company determined that it was the primary beneficiary and has a controlling financial interest in this particular K-Series securitization (see Note 6).

[2] The Company classified the multi-family CMBS issued by two K-Series securitizations and held by the Financing VIE as available for sale securities as the purpose is not to trade these securities. The Financing VIE consolidated one K-Series securitization that issued certain of the multi-family CMBS owned by the Company, including its assets, liabilities, income and expenses, in its financial statements, as based on a number of factors, the Company determined that it was the primary beneficiary and has a controlling financial interest in this particular K-Series securitization (see Note 6).

[3] Our consolidated balance sheets include assets and liabilities of consolidated variable interest entities ("VIEs") as the Company is the primary beneficiary of these VIEs. As of December 31, 2015 and December 31, 2014, assets of consolidated VIEs totaled $7,413,082 and $8,847,078, respectively, and the liabilities of consolidated VIEs totaled $7,078,162 and $8,457,034, respectively. See Note 7 for further discussion.

[4] In December 2015, the Company repaid the Company’s outstanding notes from its distressed residential mortgage loan securitization transaction completed in December 2012 with an original principal amount of $38.7 million and outstanding principal balance at the time of repayment amounting to $5.5 million. With the repayment of the notes, the Company terminated and deconsolidated the Financing VIE that facilitated this financing transaction and the distressed residential loans serving as collateral on the notes were transferred back to the Company.

[5] Classified as securitized debt in the liability section of the Company’s accompanying consolidated balance sheets.

[6] The Company engaged in the re-securitization transaction primarily for the purpose of obtaining non-recourse financing on a portion of its multi-family CMBS portfolio. As a result of engaging in this transaction, the Company remains economically exposed to the first loss position on the underlying multi-family CMBS transferred to the Consolidated VIE. The holders of the Note have no recourse to the general credit of the Company, but the Company does have the obligation, under certain circumstances, to repurchase assets upon the breach of certain representations and warranties. The Company will receive all remaining cash flow, if any, through its retained ownership.

[7] The multi-family CMBS serving as collateral under the November 2013 collateralized recourse financing are comprised of securities issued from three separate Freddie Mac-sponsored multi-family K-Series securitizations. The Financing VIE consolidated these K-Series securitizations, including their assets, liabilities, income and expenses, in its financial statements as based on a number of factors, the Company determined that it was the primary beneficiary and has a controlling financial interest in such K-Series securitizations (see Note 6).

[8] The multi-family CMBS serving as collateral under the November 2013 collateralized recourse financing are comprised of securities issued from three separate Freddie Mac-sponsored multifamily K-Series securitizations. The Financing VIE consolidated these K-Series securitizations, including their assets, liabilities and expenses, in its financial statements as based on a number of factors, the Company determined that it was the primary beneficiary and has a controlling financial interest in such K-Series securitizations (see Note 6). In September 2014, the Company repaid the Company’s outstanding notes from its collateralized recourse financing transaction completed in November 2012 with a principal amount of $52.0 million. With the repayment of the notes, the Company terminated and deconsolidated the Financing VIE that facilitated this financing transaction and the multi-family CMBS serving as collateral on the notes were transferred back to the Company.

[9] The Company entered into a CMBS Master Repurchase Agreements with a three-year term for the purpose of financing a portion of its multi-family CMBS portfolio. In connection with the transaction, the Company agreed to guarantee the due and punctual payment of its wholly-owned subsidiary's obligations under the CMBS Master Repurchase Agreement.

[10] The Company engaged in these transactions for the purpose of financing distressed residential mortgage loans acquired by the Company. The distressed residential mortgage loans serving as collateral for the financings are comprised of performing, re-performing and to a lesser extent non-performing, fixed and adjustable-rate, fully-amortizing, interest only and balloon, seasoned mortgage loans secured by first liens on one to four family properties. Two of the four securitization transactions provide for a revolving period of one to two years from the date of the respective financing (“Revolving Period”) where no principal payments will be made on the note. All cash proceeds generated by the distressed residential mortgage loans and received by the respective securitization trust during the Revolving Period, after payment of interest on the note, reserve amounts and certain other transaction expenses, will be available for the purchase by the trust of additional mortgage loans that satisfy certain eligibility criteria. In December 2015, the Company repaid the Company’s outstanding notes from its distressed residential mortgage loan securitization transaction completed in December 2012 with an original principal amount of $38.7 million and outstanding principal balance at the time of repayment amounting to $5.5 million.. With the repayment of the notes, the Company terminated and deconsolidated the Financing VIE that facilitated this financing transaction and the distressed residential loans serving as collateral on the notes were transferred back to the Company.

[11] In February 2015, the Company sold a first loss tranche PO security issued by one of the Consolidated K-Series securitizations obtaining total proceeds of approximately $44.3 million and realizing a gain of approximately $1.5 million. The sale resulted in a de-consolidation of $1.1 billion in Multi-Family loans held in a securitization trust and $1.0 billion in Multi-Family CDOs.

[12] Two of the Company's Freddie Mac-sponsored multi-family K-Series securitizations included in the Consolidated K-Series are not subject to any Financing VIE as of December 31, 2014.