Segment Reporting - Summary of Financial Information (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2023	Dec. 31, 2022	Dec. 31, 2021
Segment Reporting Information [Line Items]
Interest Income	$ 605,597	$ 482,419	$ 362,302
Interest Expense	429,118	258,843	120,385
Net Interest Income/(Expense)	176,479	223,576	241,917
Reversal of Provision/(Provision) for Credit Losses on Residential Whole Loans	8,853	2,646	44,863
Provision for Credit Losses on Other Assets	0	(28,579)	0
Net Interest Income after Provision for Credit Losses	185,332	197,643	286,780
Net gain on residential whole loans measured at fair value through earnings	89,850	(866,762)	16,243
Impairment and other net gain/(loss) on securities and other portfolio investments	6,225	(25,067)	74,496
Net gain on real estate owned	9,392	25,379	22,838
Net gain/(loss) on derivatives used for risk management purposes	3,761	255,179	1,426
Net loss on securitized debt measured at fair value through earnings	(99,589)	290,639	15,027
Lima One - origination, servicing and other fee income	43,384	46,745	22,600
Net realized loss on residential whole loans held at carrying value	(1,240)	0	0
Other, net	11,331	8,623	9,647
Other Income/(Loss), net	63,114	(265,264)	162,277
Compensation and benefits	85,799	76,728	53,817
General and administrative expenses	44,147	35,138	28,903
Loan servicing, financing and other related costs	34,136	42,894	30,867
Amortization of intangible assets	4,200	9,200	6,600
Net Income/(Loss)	80,164	(231,581)	328,870
Less Preferred Stock Dividend Requirement	32,875	32,875	32,875
Net Income/(Loss) Available to Common Stock and Participating Securities	47,289	(264,456)	295,995
Total Assets	10,772,690	9,112,405
Operating Segments | Mortgage-Related Assets
Segment Reporting Information [Line Items]
Interest Income	364,081	364,761	347,863
Interest Expense	249,458	176,725	99,905
Net Interest Income/(Expense)	114,623	188,036	247,958
Reversal of Provision/(Provision) for Credit Losses on Residential Whole Loans	8,539	2,842	44,981
Provision for Credit Losses on Other Assets		0
Net Interest Income after Provision for Credit Losses	123,162	190,878	292,939
Net gain on residential whole loans measured at fair value through earnings	69,486	(730,028)	(2,719)
Impairment and other net gain/(loss) on securities and other portfolio investments	8,073	(3,146)	1,607
Net gain on real estate owned	9,274	25,348	22,760
Net gain/(loss) on derivatives used for risk management purposes	839	217,961	1,457
Net loss on securitized debt measured at fair value through earnings	(66,969)	231,176	14,594
Lima One - origination, servicing and other fee income	0	0	0
Net realized loss on residential whole loans held at carrying value	(1,240)
Other, net	7,960	4,282	759
Other Income/(Loss), net	27,423	(254,407)	38,458
Compensation and benefits	0	0	0
General and administrative expenses	214	0	0
Loan servicing, financing and other related costs	20,100	25,384	25,250
Amortization of intangible assets	0	0	0
Net Income/(Loss)	130,271	(88,913)	306,147
Less Preferred Stock Dividend Requirement	0	0	0
Net Income/(Loss) Available to Common Stock and Participating Securities	130,271	(88,913)	306,147
Total Assets	6,370,237	6,065,557
Operating Segments | Lima One
Segment Reporting Information [Line Items]
Interest Income	228,825	113,134	14,249
Interest Expense	164,059	66,358	4,691
Net Interest Income/(Expense)	64,766	46,776	9,558
Reversal of Provision/(Provision) for Credit Losses on Residential Whole Loans	314	(196)	(118)
Provision for Credit Losses on Other Assets		0
Net Interest Income after Provision for Credit Losses	65,080	46,580	9,440
Net gain on residential whole loans measured at fair value through earnings	20,364	(136,734)	18,962
Impairment and other net gain/(loss) on securities and other portfolio investments	0	0	0
Net gain on real estate owned	118	31	78
Net gain/(loss) on derivatives used for risk management purposes	2,922	37,218	(31)
Net loss on securitized debt measured at fair value through earnings	(32,620)	59,463	433
Lima One - origination, servicing and other fee income	43,384	46,745	22,600
Net realized loss on residential whole loans held at carrying value	0
Other, net	2,284	537	128
Other Income/(Loss), net	36,452	7,260	42,170
Compensation and benefits	44,827	39,241	18,130
General and administrative expenses	17,537	13,944	6,010
Loan servicing, financing and other related costs	1,515	1,120	436
Amortization of intangible assets	4,200	9,200	6,600
Net Income/(Loss)	33,453	(9,665)	20,434
Less Preferred Stock Dividend Requirement	0	0	0
Net Income/(Loss) Available to Common Stock and Participating Securities	33,453	(9,665)	20,434
Total Assets	4,000,932	2,618,695
Corporate
Segment Reporting Information [Line Items]
Interest Income	12,691	4,524	190
Interest Expense	15,601	15,760	15,789
Net Interest Income/(Expense)	(2,910)	(11,236)	(15,599)
Reversal of Provision/(Provision) for Credit Losses on Residential Whole Loans	0	0	0
Provision for Credit Losses on Other Assets		(28,579)
Net Interest Income after Provision for Credit Losses	(2,910)	(39,815)	(15,599)
Net gain on residential whole loans measured at fair value through earnings	0	0	0
Impairment and other net gain/(loss) on securities and other portfolio investments	(1,848)	(21,921)	72,889
Net gain on real estate owned	0	0	0
Net gain/(loss) on derivatives used for risk management purposes	0	0	0
Net loss on securitized debt measured at fair value through earnings	0	0	0
Lima One - origination, servicing and other fee income	0	0	0
Net realized loss on residential whole loans held at carrying value	0
Other, net	1,087	3,804	8,760
Other Income/(Loss), net	(761)	(18,117)	81,649
Compensation and benefits	40,972	37,487	35,687
General and administrative expenses	26,396	21,194	22,893
Loan servicing, financing and other related costs	12,521	16,390	5,181
Amortization of intangible assets	0	0	0
Net Income/(Loss)	(83,560)	(133,003)	2,289
Less Preferred Stock Dividend Requirement	32,875	32,875	32,875
Net Income/(Loss) Available to Common Stock and Participating Securities	(116,435)	(165,878)	$ (30,586)
Total Assets	$ 401,521	$ 428,153