Use of Special Purpose Entities (SPE) and Variable Interest Entities (VIE) - Classification and Carrying Value of Unconsolidated VIEs (Details) - Variable Interest Entity, Not Primary Beneficiary - USD ($) $ in Thousands	Sep. 30, 2019	Dec. 31, 2018
Variable Interest Entity [Line Items]
Total assets	$ 381,714	$ 269,753
Investment securities, available for sale, at fair value, held in securitization trusts
Variable Interest Entity [Line Items]
Total assets	48,254	52,700
Receivables and other assets
Variable Interest Entity [Line Items]
Total assets		72
Preferred equity and mezzanine loan investments
Variable Interest Entity [Line Items]
Total assets	178,997	165,555
Investments in unconsolidated entities
Variable Interest Entity [Line Items]
Total assets	154,463	51,426
ABS
Variable Interest Entity [Line Items]
Total assets	48,254
ABS | Investment securities, available for sale, at fair value, held in securitization trusts
Variable Interest Entity [Line Items]
Total assets	48,254
ABS | Preferred equity and mezzanine loan investments
Variable Interest Entity [Line Items]
Total assets	0
ABS | Investments in unconsolidated entities
Variable Interest Entity [Line Items]
Total assets	0
Preferred equity investments in multi-family properties
Variable Interest Entity [Line Items]
Total assets	265,510	195,101
Preferred equity investments in multi-family properties | Investment securities, available for sale, at fair value, held in securitization trusts
Variable Interest Entity [Line Items]
Total assets	0	0
Preferred equity investments in multi-family properties | Receivables and other assets
Variable Interest Entity [Line Items]
Total assets		0
Preferred equity investments in multi-family properties | Preferred equity and mezzanine loan investments
Variable Interest Entity [Line Items]
Total assets	172,826	154,629
Preferred equity investments in multi-family properties | Investments in unconsolidated entities
Variable Interest Entity [Line Items]
Total assets	92,684	40,472
Multi-family CMBS
Variable Interest Entity [Line Items]
Total assets		52,772
Multi-family CMBS | Investment securities, available for sale, at fair value, held in securitization trusts
Variable Interest Entity [Line Items]
Total assets		52,700
Multi-family CMBS | Receivables and other assets
Variable Interest Entity [Line Items]
Total assets		72
Multi-family CMBS | Preferred equity and mezzanine loan investments
Variable Interest Entity [Line Items]
Total assets		0
Multi-family CMBS | Investments in unconsolidated entities
Variable Interest Entity [Line Items]
Total assets		0
Mezzanine loans on multi-family properties
Variable Interest Entity [Line Items]
Total assets	6,171	10,926
Mezzanine loans on multi-family properties | Investment securities, available for sale, at fair value, held in securitization trusts
Variable Interest Entity [Line Items]
Total assets	0	0
Mezzanine loans on multi-family properties | Receivables and other assets
Variable Interest Entity [Line Items]
Total assets		0
Mezzanine loans on multi-family properties | Preferred equity and mezzanine loan investments
Variable Interest Entity [Line Items]
Total assets	6,171	10,926
Mezzanine loans on multi-family properties | Investments in unconsolidated entities
Variable Interest Entity [Line Items]
Total assets	0	0
Equity investments in entities that invest in residential properties
Variable Interest Entity [Line Items]
Total assets	61,779	10,954
Equity investments in entities that invest in residential properties | Investment securities, available for sale, at fair value, held in securitization trusts
Variable Interest Entity [Line Items]
Total assets	0	0
Equity investments in entities that invest in residential properties | Receivables and other assets
Variable Interest Entity [Line Items]
Total assets		0
Equity investments in entities that invest in residential properties | Preferred equity and mezzanine loan investments
Variable Interest Entity [Line Items]
Total assets	0	0
Equity investments in entities that invest in residential properties | Investments in unconsolidated entities
Variable Interest Entity [Line Items]
Total assets	$ 61,779	$ 10,954