Investment Securities - Summary (Details) - USD ($) $ in Thousands	Dec. 31, 2021	Dec. 31, 2020
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	$ (609,251)	$ (178,291)
Senior
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	(4,089)	(11,936)
Mezzanine
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	(1,631)	(1,633)
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	0	(15,104)
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	0	0
AOMT RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	(5,720)	(28,673)
Senior
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt		0
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	0	0
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	0
Other Non-Agency RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	0	0
Fannie Mae
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	(267,286)
Freddie Mac
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	(87,495)
Whole Pool Agency RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	(354,781)
RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	(360,501)	(28,673)
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	0	0
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	0	0
AOMT CMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Repurchase Debt	0	0
Senior
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	3,076	11,477
Allocated Capital	(1,013)	(459)
Mezzanine
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	2,178	2,207
Allocated Capital	547	574
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	80,058	78,806
Allocated Capital	80,058	63,702
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	15,052	31,842
Allocated Capital	15,052	31,842
AOMT RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	100,364	124,332
Allocated Capital	94,644	95,659
Senior
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value		6,820
Allocated Capital		6,820
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	10,292	18,784
Allocated Capital	10,292	18,784
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	2,923
Allocated Capital	2,923
Other Non-Agency RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	13,215	25,604
Allocated Capital	13,215	25,604
Fannie Mae
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	281,225
Allocated Capital	13,939
Freddie Mac
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	90,830
Allocated Capital	3,335
Whole Pool Agency RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	372,055
Allocated Capital	17,274
RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	485,634	149,936
Repurchase Debt	(360,501)	(28,673)
Allocated Capital	125,133	121,263
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	7,993	5,766
Allocated Capital	7,993	5,766
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	2,763	3,030
Allocated Capital	2,763	3,030
AOMT CMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	10,756	8,796
Allocated Capital	$ 10,756	$ 8,796