Use of Special Purpose Entities (SPE) and Variable Interest Entities (VIE) - Classification and Carrying Value of Unconsolidated VIEs (Details) - Equity Method Investment, Nonconsolidated Investee or Group of Investees - USD ($) $ in Thousands	Mar. 31, 2024	Dec. 31, 2023
Maximum exposure
Variable Interest Entity [Line Items]
Investments	$ 218,238	$ 230,216
Maximum exposure | Multi-family loans
Variable Interest Entity [Line Items]
Investments	91,905	95,792
Maximum exposure | Investment securities available for sale, at fair value
Variable Interest Entity [Line Items]
Investments	23,855	24,462
Maximum exposure | Equity investments
Variable Interest Entity [Line Items]
Investments	102,478	109,962
Non-Agency RMBS
Variable Interest Entity [Line Items]
Investments	23,855	24,462
Non-Agency RMBS | Multi-family loans
Variable Interest Entity [Line Items]
Investments	0	0
Non-Agency RMBS | Investment securities available for sale, at fair value
Variable Interest Entity [Line Items]
Investments	23,855	24,462
Non-Agency RMBS | Equity investments
Variable Interest Entity [Line Items]
Investments	0	0
Preferred equity investments in multi-family properties
Variable Interest Entity [Line Items]
Investments	192,713	200,034
Preferred equity investments in multi-family properties | Multi-family loans
Variable Interest Entity [Line Items]
Investments	91,905	95,792
Preferred equity investments in multi-family properties | Investment securities available for sale, at fair value
Variable Interest Entity [Line Items]
Investments	0	0
Preferred equity investments in multi-family properties | Equity investments
Variable Interest Entity [Line Items]
Investments	100,808	104,242
Joint venture equity investments in multi-family properties
Variable Interest Entity [Line Items]
Investments	1,670	5,720
Joint venture equity investments in multi-family properties | Multi-family loans
Variable Interest Entity [Line Items]
Investments	0	0
Joint venture equity investments in multi-family properties | Investment securities available for sale, at fair value
Variable Interest Entity [Line Items]
Investments	0	0
Joint venture equity investments in multi-family properties | Equity investments
Variable Interest Entity [Line Items]
Investments	$ 1,670	$ 5,720