Segment Reporting (Details) - USD ($) $ in Thousands	3 Months Ended				6 Months Ended
	Jun. 30, 2023	Mar. 31, 2023	Jun. 30, 2022	Mar. 31, 2022	Jun. 30, 2023	Jun. 30, 2022	Dec. 31, 2022
Segment Reporting Information [Line Items]
Interest Income	$ 144,350		$ 109,321		$ 276,555	$ 215,670
Interest Expense	99,845		56,742		192,681	100,038
Net Interest Income	44,505		52,579		83,874	115,632
(Provision)/Reversal of Provision for Credit Losses on Residential Whole Loans	(294)		(1,817)		(281)	1,694
Provision for Credit Losses on Other Assets	0		(28,579)		0	(28,579)
Net Interest Income/(Expense) after Provision for Credit Losses	44,211		22,183		83,593	88,747
Net loss on residential whole loans measured at fair value through earnings	(130,703)		(218,181)		(1,529)	(506,116)
Impairment and other net loss on securities and other portfolio investments	(4,569)		(12,046)		(1,638)	(15,747)
Net gain/(loss) on real estate owned	2,153		7,185		6,095	15,917
Net gain on derivatives used for risk management purposes	60,451		47,804		39,243	141,905
Net gain on securitized debt measured at fair value through earnings	27,394		84,573		(24,331)	148,690
Lima One - origination, servicing and other fee income	11,477		10,673		20,453	25,167
Other, net	5,496		3,544		8,668	6,220
Other (Loss)/Income, net	(28,301)		(76,448)		46,961	(183,964)
General and administrative expenses (including compensation)	32,940		29,567		63,961	57,820
Loan servicing, financing, and other related costs	7,598		13,235		17,137	23,636
Amortization of intangible assets	1,300		3,300		2,600	6,600
Net (Loss)/Income	(25,928)	$ 72,784	(100,367)	$ (82,906)	46,856	(183,273)
Less Preferred Stock Dividend Requirement	8,218		8,219		16,437	16,438
Net (Loss)/Income Available to Common Stock and Participating Securities	(34,146)		(108,586)		30,419	(199,711)
Total Assets	9,736,231				9,736,231		$ 9,112,405
Mortgage-Related Assets
Segment Reporting Information [Line Items]
Interest Income	89,912		84,732		174,732	173,554
Interest Expense	58,940		39,889		116,017	72,898
Net Interest Income	30,972		44,843		58,715	100,656
(Provision)/Reversal of Provision for Credit Losses on Residential Whole Loans	(294)		(1,785)		(595)	1,731
Provision for Credit Losses on Other Assets			0			0
Net Interest Income/(Expense) after Provision for Credit Losses	30,678		43,058		58,120	102,387
Net loss on residential whole loans measured at fair value through earnings	(97,459)		(177,203)		(1,949)	(430,330)
Impairment and other net loss on securities and other portfolio investments	(3,697)		(1,463)		(767)	(4,384)
Net gain/(loss) on real estate owned	2,493		7,150		6,417	15,885
Net gain on derivatives used for risk management purposes	45,142		44,161		28,820	130,396
Net gain on securitized debt measured at fair value through earnings	18,887		64,966		(15,933)	122,545
Lima One - origination, servicing and other fee income	0		0		0	0
Other, net	3,812		2,598		6,020	3,917
Other (Loss)/Income, net	(30,822)		(59,791)		22,608	(161,971)
General and administrative expenses (including compensation)	0		0		0	0
Loan servicing, financing, and other related costs	5,395		6,513		10,114	13,445
Amortization of intangible assets	0		0		0	0
Net (Loss)/Income	(5,539)		(23,246)		70,614	(73,029)
Less Preferred Stock Dividend Requirement	0		0		0	0
Net (Loss)/Income Available to Common Stock and Participating Securities	(5,539)		(23,246)		70,614	(73,029)
Total Assets	6,183,728				6,183,728		6,065,557
Lima One
Segment Reporting Information [Line Items]
Interest Income	51,308		24,353		95,828	41,853
Interest Expense	36,943		12,916		68,747	19,272
Net Interest Income	14,365		11,437		27,081	22,581
(Provision)/Reversal of Provision for Credit Losses on Residential Whole Loans	0		(32)		314	(37)
Provision for Credit Losses on Other Assets			0			0
Net Interest Income/(Expense) after Provision for Credit Losses	14,365		11,405		27,395	22,544
Net loss on residential whole loans measured at fair value through earnings	(33,244)		(40,978)		420	(75,786)
Impairment and other net loss on securities and other portfolio investments	0		0		0	0
Net gain/(loss) on real estate owned	(340)		35		(322)	32
Net gain on derivatives used for risk management purposes	15,309		3,643		10,423	11,509
Net gain on securitized debt measured at fair value through earnings	8,507		19,607		(8,398)	26,145
Lima One - origination, servicing and other fee income	11,477		10,673		20,453	25,167
Other, net	1,076		(26)		1,447	28
Other (Loss)/Income, net	2,785		(7,046)		24,023	(12,905)
General and administrative expenses (including compensation)	15,601		13,013		28,136	25,232
Loan servicing, financing, and other related costs	131		323		350	559
Amortization of intangible assets	1,300		3,300		2,600	6,600
Net (Loss)/Income	118		(12,277)		20,332	(22,752)
Less Preferred Stock Dividend Requirement	0		0		0	0
Net (Loss)/Income Available to Common Stock and Participating Securities	118		(12,277)		20,332	(22,752)
Total Assets	3,156,741				3,156,741		2,618,695
Corporate
Segment Reporting Information [Line Items]
Interest Income	3,130		236		5,995	263
Interest Expense	3,962		3,937		7,917	7,868
Net Interest Income	(832)		(3,701)		(1,922)	(7,605)
(Provision)/Reversal of Provision for Credit Losses on Residential Whole Loans	0		0		0	0
Provision for Credit Losses on Other Assets			(28,579)			(28,579)
Net Interest Income/(Expense) after Provision for Credit Losses	(832)		(32,280)		(1,922)	(36,184)
Net loss on residential whole loans measured at fair value through earnings	0		0		0	0
Impairment and other net loss on securities and other portfolio investments	(872)		(10,583)		(871)	(11,363)
Net gain/(loss) on real estate owned	0		0		0	0
Net gain on derivatives used for risk management purposes	0		0		0	0
Net gain on securitized debt measured at fair value through earnings	0		0		0	0
Lima One - origination, servicing and other fee income	0		0		0	0
Other, net	608		972		1,201	2,275
Other (Loss)/Income, net	(264)		(9,611)		330	(9,088)
General and administrative expenses (including compensation)	17,339		16,554		35,825	32,588
Loan servicing, financing, and other related costs	2,072		6,399		6,673	9,632
Amortization of intangible assets	0		0		0	0
Net (Loss)/Income	(20,507)		(64,844)		(44,090)	(87,492)
Less Preferred Stock Dividend Requirement	8,218		8,219		16,437	16,438
Net (Loss)/Income Available to Common Stock and Participating Securities	(28,725)		$ (73,063)		(60,527)	$ (103,930)
Total Assets	$ 395,762				$ 395,762		$ 428,153