Segment Reporting (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2021	Dec. 31, 2020	Dec. 31, 2019
Segment Reporting Information [Line Items]
Interest Income	$ 362,302	$ 432,832	$ 695,907
Interest Expense	120,385	268,758	332,356
Net Interest Income	241,917	164,074	363,551
Provision for Credit Losses on Residential Whole Loans	44,863
Provision for Credit Losses on Other Assets	0
Net Interest Income after Provision for Credit and Valuation Losses	286,780	141,693	360,982
Net mark-to-market and other net loss on residential whole loans measured at fair value	16,736	20,765	44,149
Net gain on derivatives used for risk management purposes	1,426
Net mark-to-market on Securitized debt measured at fair value	15,027
Net gain/(loss) on real estate owned	22,838	5,391	(5,878)
Gain on investment in lima one common equity	38,933	0	0
Impairment and other gains and losses on securities available-for-sale and other assets	33,956	(425,082)	(180)
Lima One - origination, servicing and other fee income	22,600	0	0
Other (loss)/income, net	13,587
Other Income/(Loss), net	165,103	(679,569)	111,676
General and administrative expenses (including compensation)	85,546
Loan servicing, financing and other related costs	30,867	40,372	41,893
Amortization of intangible assets	6,600	0	0
Net Income/(Loss)	328,870	(679,390)	378,117
Less Preferred Stock Dividend Requirement	32,875	29,796	15,000
Net Income/(Loss) Available to Common Stock and Participating Securities	295,995	(709,186)	$ 363,117
Total Assets	9,139,688	$ 6,932,300
Mortgage-Related Assets
Segment Reporting Information [Line Items]
Interest Income	347,863
Interest Expense	99,905
Net Interest Income	247,958
Provision for Credit Losses on Residential Whole Loans	44,981
Provision for Credit Losses on Other Assets	0
Net Interest Income after Provision for Credit and Valuation Losses	292,939
Net mark-to-market and other net loss on residential whole loans measured at fair value	(2,047)
Net gain on derivatives used for risk management purposes	1,457
Net mark-to-market on Securitized debt measured at fair value	14,594
Net gain/(loss) on real estate owned	22,760
Gain on investment in lima one common equity	0
Impairment and other gains and losses on securities available-for-sale and other assets	0
Lima One - origination, servicing and other fee income	0
Other (loss)/income, net	1,694
Other Income/(Loss), net	38,458
General and administrative expenses (including compensation)	0
Loan servicing, financing and other related costs	25,250
Amortization of intangible assets	0
Net Income/(Loss)	306,147
Less Preferred Stock Dividend Requirement	0
Net Income/(Loss) Available to Common Stock and Participating Securities	306,147
Total Assets	7,567,084
Lima One
Segment Reporting Information [Line Items]
Interest Income	14,249
Interest Expense	4,691
Net Interest Income	9,558
Provision for Credit Losses on Residential Whole Loans	(118)
Provision for Credit Losses on Other Assets	0
Net Interest Income after Provision for Credit and Valuation Losses	9,440
Net mark-to-market and other net loss on residential whole loans measured at fair value	18,783
Net gain on derivatives used for risk management purposes	(31)
Net mark-to-market on Securitized debt measured at fair value	433
Net gain/(loss) on real estate owned	78
Gain on investment in lima one common equity	0
Impairment and other gains and losses on securities available-for-sale and other assets	0
Lima One - origination, servicing and other fee income	22,600
Other (loss)/income, net	307
Other Income/(Loss), net	42,170
General and administrative expenses (including compensation)	24,140
Loan servicing, financing and other related costs	436
Amortization of intangible assets	6,600
Net Income/(Loss)	20,434
Less Preferred Stock Dividend Requirement	0
Net Income/(Loss) Available to Common Stock and Participating Securities	20,434
Total Assets	1,200,737
Corporate
Segment Reporting Information [Line Items]
Interest Income	190
Interest Expense	15,789
Net Interest Income	(15,599)
Provision for Credit Losses on Residential Whole Loans	0
Provision for Credit Losses on Other Assets	0
Net Interest Income after Provision for Credit and Valuation Losses	(15,599)
Net mark-to-market and other net loss on residential whole loans measured at fair value	0
Net gain on derivatives used for risk management purposes	0
Net mark-to-market on Securitized debt measured at fair value	0
Net gain/(loss) on real estate owned	0
Gain on investment in lima one common equity	38,933
Impairment and other gains and losses on securities available-for-sale and other assets	33,956
Lima One - origination, servicing and other fee income	0
Other (loss)/income, net	11,586
Other Income/(Loss), net	84,475
General and administrative expenses (including compensation)	61,406
Loan servicing, financing and other related costs	5,181
Amortization of intangible assets	0
Net Income/(Loss)	2,289
Less Preferred Stock Dividend Requirement	32,875
Net Income/(Loss) Available to Common Stock and Participating Securities	(30,586)
Total Assets	$ 371,867