Fair Value of Financial Instruments - Assets and Liabilities Measured at Fair Value on a Recurring Basis (Details) - USD ($) $ in Thousands	Jun. 30, 2021	Dec. 31, 2020
Assets carried at fair value
Investment securities available for sale, at fair value	$ 225,267	$ 311,284
Fair Value, Measurements, Recurring
Assets carried at fair value
Equity investments	204,697	259,095
Total	4,141,566	4,196,580
Liabilities carried at fair value
Total	948,090	1,054,335
Fair Value, Measurements, Recurring | Residential loans
Assets carried at fair value
Residential loans, at fair value	1,226,841	1,090,930
Fair Value, Measurements, Recurring | Residential loans | Consolidated SLST | VIE, Primary Beneficiary
Assets carried at fair value
Residential loans, at fair value	1,176,539	1,266,785
Fair Value, Measurements, Recurring | Residential loans held in securitization trusts
Assets carried at fair value
Residential loans, at fair value	799,363	691,451
Fair Value, Measurements, Recurring | Multi-family loans
Assets carried at fair value
Multi-family loans	126,719	163,593
Fair Value, Measurements, Recurring | Agency RMBS
Assets carried at fair value
Investment securities available for sale, at fair value	131,299	139,395
Fair Value, Measurements, Recurring | Non-Agency RMBS
Assets carried at fair value
Investment securities available for sale, at fair value	287,988	355,666
Fair Value, Measurements, Recurring | CMBS
Assets carried at fair value
Investment securities available for sale, at fair value	145,500	186,440
Fair Value, Measurements, Recurring | ABS
Assets carried at fair value
Investment securities available for sale, at fair value	42,620	43,225
Fair Value, Measurements, Recurring | Residential Collateralized Debt Obligations, at Fair Value | Consolidated SLST | VIE, Primary Beneficiary
Liabilities carried at fair value
Collateralized debt obligations, at fair value	948,090	1,054,335
Level 1 | Fair Value, Measurements, Recurring
Assets carried at fair value
Equity investments	0	0
Total	0	0
Liabilities carried at fair value
Total	0	0
Level 1 | Fair Value, Measurements, Recurring | Residential loans
Assets carried at fair value
Residential loans, at fair value	0	0
Level 1 | Fair Value, Measurements, Recurring | Residential loans | Consolidated SLST | VIE, Primary Beneficiary
Assets carried at fair value
Residential loans, at fair value	0	0
Level 1 | Fair Value, Measurements, Recurring | Residential loans held in securitization trusts
Assets carried at fair value
Residential loans, at fair value	0	0
Level 1 | Fair Value, Measurements, Recurring | Multi-family loans
Assets carried at fair value
Multi-family loans	0	0
Level 1 | Fair Value, Measurements, Recurring | Agency RMBS
Assets carried at fair value
Investment securities available for sale, at fair value	0	0
Level 1 | Fair Value, Measurements, Recurring | Non-Agency RMBS
Assets carried at fair value
Investment securities available for sale, at fair value	0	0
Level 1 | Fair Value, Measurements, Recurring | CMBS
Assets carried at fair value
Investment securities available for sale, at fair value	0	0
Level 1 | Fair Value, Measurements, Recurring | ABS
Assets carried at fair value
Investment securities available for sale, at fair value	0	0
Level 1 | Fair Value, Measurements, Recurring | Residential Collateralized Debt Obligations, at Fair Value | Consolidated SLST | VIE, Primary Beneficiary
Liabilities carried at fair value
Collateralized debt obligations, at fair value	0	0
Level 2
Assets carried at fair value
Investment securities available for sale, at fair value	607,407	724,726
Level 2 | Fair Value, Measurements, Recurring
Assets carried at fair value
Equity investments	0	0
Total	607,407	724,726
Liabilities carried at fair value
Total	0	0
Level 2 | Fair Value, Measurements, Recurring | Residential loans
Assets carried at fair value
Residential loans, at fair value	0	0
Level 2 | Fair Value, Measurements, Recurring | Residential loans | Consolidated SLST | VIE, Primary Beneficiary
Assets carried at fair value
Residential loans, at fair value	0	0
Level 2 | Fair Value, Measurements, Recurring | Residential loans held in securitization trusts
Assets carried at fair value
Residential loans, at fair value	0	0
Level 2 | Fair Value, Measurements, Recurring | Multi-family loans
Assets carried at fair value
Multi-family loans	0	0
Level 2 | Fair Value, Measurements, Recurring | Agency RMBS
Assets carried at fair value
Investment securities available for sale, at fair value	131,299	139,395
Level 2 | Fair Value, Measurements, Recurring | Non-Agency RMBS
Assets carried at fair value
Investment securities available for sale, at fair value	287,988	355,666
Level 2 | Fair Value, Measurements, Recurring | CMBS
Assets carried at fair value
Investment securities available for sale, at fair value	145,500	186,440
Level 2 | Fair Value, Measurements, Recurring | ABS
Assets carried at fair value
Investment securities available for sale, at fair value	42,620	43,225
Level 2 | Fair Value, Measurements, Recurring | Residential Collateralized Debt Obligations, at Fair Value | Consolidated SLST | VIE, Primary Beneficiary
Liabilities carried at fair value
Collateralized debt obligations, at fair value	0	0
Level 3
Assets carried at fair value
Equity investments	204,697	259,095
Level 3 | Fair Value, Measurements, Recurring
Assets carried at fair value
Equity investments	204,697	259,095
Total	3,534,159	3,471,854
Liabilities carried at fair value
Total	948,090	1,054,335
Level 3 | Fair Value, Measurements, Recurring | Residential loans
Assets carried at fair value
Residential loans, at fair value	1,226,841	1,090,930
Level 3 | Fair Value, Measurements, Recurring | Residential loans | Consolidated SLST | VIE, Primary Beneficiary
Assets carried at fair value
Residential loans, at fair value	1,176,539	1,266,785
Level 3 | Fair Value, Measurements, Recurring | Residential loans held in securitization trusts
Assets carried at fair value
Residential loans, at fair value	799,363	691,451
Level 3 | Fair Value, Measurements, Recurring | Multi-family loans
Assets carried at fair value
Multi-family loans	126,719	163,593
Level 3 | Fair Value, Measurements, Recurring | Agency RMBS
Assets carried at fair value
Investment securities available for sale, at fair value	0	0
Level 3 | Fair Value, Measurements, Recurring | Non-Agency RMBS
Assets carried at fair value
Investment securities available for sale, at fair value	0	0
Level 3 | Fair Value, Measurements, Recurring | CMBS
Assets carried at fair value
Investment securities available for sale, at fair value	0	0
Level 3 | Fair Value, Measurements, Recurring | ABS
Assets carried at fair value
Investment securities available for sale, at fair value	0	0
Level 3 | Fair Value, Measurements, Recurring | Residential Collateralized Debt Obligations, at Fair Value | Consolidated SLST | VIE, Primary Beneficiary
Liabilities carried at fair value
Collateralized debt obligations, at fair value	$ 948,090	$ 1,054,335