Use of Special Purpose Entities (SPE) and Variable Interest Entities (VIE) - Classification and Carrying Value of Unconsolidated VIEs (Details) $ in Thousands	Dec. 31, 2018 USD ($) securitization	Dec. 31, 2017 USD ($)
Variable Interest Entity [Line Items]
Unconsolidated VIE, maximum loss exposure	$ 269,800	$ 220,800
Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	269,753	220,797
Investment securities available for sale, at fair value, held in securitization trusts | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	52,700	47,922
Receivables and other Assets | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	72	73
Preferred equity and mezzanine loan investments | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	165,555	138,920
Investment in unconsolidated entities | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	51,426	33,882
Other VIEs, Multi-family CMBS | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	52,772	47,995
Other VIEs, Multi-family CMBS | Investment securities available for sale, at fair value, held in securitization trusts | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	52,700	47,922
Other VIEs, Multi-family CMBS | Receivables and other Assets | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	72	73
Other VIEs, Multi-family CMBS | Preferred equity and mezzanine loan investments | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	0	0
Other VIEs, Multi-family CMBS | Investment in unconsolidated entities | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	0	0
Preferred Equity Investment on Multi-family Properties | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	195,101	140,329
Preferred Equity Investment on Multi-family Properties | Investment securities available for sale, at fair value, held in securitization trusts | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	0	0
Preferred Equity Investment on Multi-family Properties | Receivables and other Assets | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	0	0
Preferred Equity Investment on Multi-family Properties | Preferred equity and mezzanine loan investments | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	154,629	132,009
Preferred Equity Investment on Multi-family Properties | Investment in unconsolidated entities | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	40,472	8,320
Mezzanine/Construction Loan on Multi-family Properties | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	10,926	6,911
Mezzanine/Construction Loan on Multi-family Properties | Investment securities available for sale, at fair value, held in securitization trusts | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	0	0
Mezzanine/Construction Loan on Multi-family Properties | Receivables and other Assets | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	0	0
Mezzanine/Construction Loan on Multi-family Properties | Preferred equity and mezzanine loan investments | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	10,926	6,911
Mezzanine/Construction Loan on Multi-family Properties | Investment in unconsolidated entities | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	0	0
Equity Investment in Entities that Invest in Multi-family Properties | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	10,954	25,562
Equity Investment in Entities that Invest in Multi-family Properties | Investment securities available for sale, at fair value, held in securitization trusts | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	0	0
Equity Investment in Entities that Invest in Multi-family Properties | Receivables and other Assets | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	0	0
Equity Investment in Entities that Invest in Multi-family Properties | Preferred equity and mezzanine loan investments | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	0	0
Equity Investment in Entities that Invest in Multi-family Properties | Investment in unconsolidated entities | Variable Interest Entity, Not Primary Beneficiary
Variable Interest Entity [Line Items]
Total Assets	$ 10,954	$ 25,562
K-Series | Multi-Family Collateralized Mortgage Backed Securities
Variable Interest Entity [Line Items]
Number of securitizations | securitization	2