Fair Value of Financial Instruments - Carrying Value and Estimated Fair Value of the Company's Financial Assets (Details) - USD ($) $ in Thousands	Jun. 30, 2023	Dec. 31, 2022	Jun. 30, 2022
Financial Assets:
Cash and cash equivalents	$ 232,497	$ 244,718	$ 407,104
Investment securities available for sale:	23,951	23,645
Equity Investments, carrying value	168,755	179,746
Equity investments in disposal group held for sale, carrying value	7,695	9,010
Derivative Asset, carrying value	2,610	2,473
Financial Liabilities:
Subordinated debentures, carrying value	45,000	45,000
Senior unsecured notes, carrying value	97,742	$ 97,384
Mortgages payable on operating real estate, carrying value	$ 544,743
Derivative Asset, Statement of Financial Position [Extensible Enumeration]	Other assets	Other assets
Residential collateralized debt obligations
Financial Liabilities:
Carrying Value	$ 1,369,632	$ 1,468,222
Residential collateralized debt obligations | VIE, Primary Beneficiary
Financial Liabilities:
Carrying Value	1,369,632	1,468,222
Consolidated SLST CDOs
Financial Liabilities:
Carrying Value	617,168	634,495
Consolidated SLST CDOs | VIE, Primary Beneficiary
Financial Liabilities:
Carrying Value	617,168	634,495
Level 1
Financial Assets:
Cash and cash equivalents	232,497	244,718
Cash and cash equivalents, estimated fair value	232,497	244,718
Level 2
Financial Assets:
Investment securities available for sale:	734,272	99,559
Derivative Asset, carrying value	2,610	2,473
Derivative Asset, estimated fair value	2,610	2,473
Derivative assets in disposal group held for sale, carrying value	20,779	29,418
Derivative assets in disposal group held for sale, estimated fair value	20,779	29,418
Financial Liabilities:
Senior unsecured notes, carrying value	97,742	97,384
Senior unsecured notes, estimated fair value	91,839	91,104
Level 2 | Portfolio Investments
Financial Liabilities:
Outstanding repurchase agreements	1,145,108	737,023
Level 3
Financial Assets:
Equity Investments, carrying value	168,755	179,746
Equity Investment, estimated fair value	168,755	179,746
Equity investments in disposal group held for sale, carrying value	7,695	9,010
Equity investments in disposal group held for sale, estimated fair value	7,695	9,010
Financial Liabilities:
Subordinated debentures, carrying value	45,000	45,000
Subordinated debentures, estimated fair value	35,829	32,721
Level 3 | Mortgage Note
Financial Liabilities:
Mortgages payable on operating real estate, carrying value	397,075	394,707
Mortgages payable on operating real estate, estimated fair value	373,616	377,327
Mortgages payable on real estate in disposal group held for sale, carrying value	734,997	865,414
Mortgages payable on real estate in disposal group held for sale, estimated fair value	733,983	864,758
Level 3 | Residential collateralized debt obligations
Financial Liabilities:
Carrying Value	1,369,632	1,468,222
Secured debt, estimated fair value	1,294,577	1,383,715
Level 3 | Consolidated SLST CDOs | VIE, Primary Beneficiary
Financial Liabilities:
Carrying Value	617,168	634,495
Secured debt, estimated fair value	617,168	634,495
Level 3 | Residential loans
Financial Assets:
Residential loans	3,136,812	3,525,080
Residential mortgage loans, estimated fair value	3,136,812	3,525,080
Level 3 | Multi-family loans
Financial Assets:
Preferred equity and mezzanine loan investments, carrying value	97,422	87,534
Preferred equity and mezzanine loan investments, estimated fair value	$ 97,422	$ 87,534