Segment Reporting (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2024	Mar. 31, 2023	Dec. 31, 2023	Dec. 31, 2022
Segment Reporting Information [Line Items]
Interest income	$ 101,520	$ 87,174
Interest expense	(70,464)	(59,617)
Noninterest Income	44,485	51,675
Operating Expenses	43,655	37,902
Income (Loss) from Continuing Operations before Equity Method Investments, Income Taxes, Noncontrolling Interest	31,886	41,330
Income tax expense (benefit)	61	21
Earnings (losses) from investments in unconsolidated entities	2,226	3,444
Net Income (Loss), Including Portion Attributable to Noncontrolling Interest	34,051	44,753
Net income (loss) attributable to non-controlling interests	482	720
Dividends on preferred stock	6,654	5,117
Net income (loss) attributable to common stockholders	26,915	38,916
Depreciation	270	358
Total Assets	15,132,600		$ 15,315,930
Liabilities	13,579,444		13,780,318
Total Equity	1,553,156	1,374,763	1,535,612	$ 1,220,886
Investment Portfolio Segment
Segment Reporting Information [Line Items]
Interest income	91,403	82,369
Interest expense	(57,310)	(52,136)
Noninterest Income	10,024	10,929
Operating Expenses	3,143	3,505
Income (Loss) from Continuing Operations before Equity Method Investments, Income Taxes, Noncontrolling Interest	40,974	37,657
Income tax expense (benefit)	0	0
Earnings (losses) from investments in unconsolidated entities	2,226	3,444
Net Income (Loss), Including Portion Attributable to Noncontrolling Interest	43,200	41,101
Net income (loss) attributable to non-controlling interests	185	238
Dividends on preferred stock	0	0
Net income (loss) attributable to common stockholders	43,015	40,863
Depreciation	0	0
Total Assets	5,620,097		5,978,988
Liabilities	4,097,762		4,555,212
Total Equity	1,522,335		1,423,776
Longbridge Segment
Segment Reporting Information [Line Items]
Interest income	8,240	2,893
Interest expense	(8,557)	(4,346)
Noninterest Income	38,170	33,398
Operating Expenses	29,099	25,447
Income (Loss) from Continuing Operations before Equity Method Investments, Income Taxes, Noncontrolling Interest	8,754	6,498
Income tax expense (benefit)	0	0
Earnings (losses) from investments in unconsolidated entities	0	0
Net Income (Loss), Including Portion Attributable to Noncontrolling Interest	8,754	6,498
Net income (loss) attributable to non-controlling interests	38	2
Dividends on preferred stock	0	0
Net income (loss) attributable to common stockholders	8,716	6,496
Depreciation	270	358
Total Assets	9,338,455		9,092,840
Liabilities	9,115,860		8,872,153
Total Equity	222,595		220,687
Corporate and Other
Segment Reporting Information [Line Items]
Interest income	1,877	1,912
Interest expense	(4,597)	(3,135)
Noninterest Income	(3,709)	7,348
Operating Expenses	11,413	8,950
Income (Loss) from Continuing Operations before Equity Method Investments, Income Taxes, Noncontrolling Interest	(17,842)	(2,825)
Income tax expense (benefit)	61	21
Earnings (losses) from investments in unconsolidated entities	0	0
Net Income (Loss), Including Portion Attributable to Noncontrolling Interest	(17,903)	(2,846)
Net income (loss) attributable to non-controlling interests	259	480
Dividends on preferred stock	6,654	5,117
Net income (loss) attributable to common stockholders	(24,816)	(8,443)
Depreciation	0	$ 0
Total Assets	174,048		244,102
Liabilities	365,822		352,953
Total Equity	$ (191,774)		$ (108,851)