Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2023	Jun. 30, 2022	Jun. 30, 2023	Jun. 30, 2022
Segment Reporting Information [Line Items]
Interest income	$ 178,981	$ 167,455	$ 357,497	$ 356,855
Interest expense	(152,885)	(126,967)	(304,964)	(263,265)
Net interest income (expense)	26,096	40,488	52,533	93,590
Non-interest income (loss)
Mortgage banking activities, net	16,552	(30,017)	33,223	(13,702)
Investment fair value changes, net	(4,596)	(87,972)	(4,723)	(94,092)
Other income, net	4,158	7,006	8,714	12,989
Realized gains, net	1,056	0	1,054	2,581
Total non-interest income (loss), net	17,170	(110,983)	38,268	(92,224)
General and administrative expenses	(30,805)	(30,199)	(66,360)	(63,475)
Portfolio management costs	(3,100)	(1,767)	(6,610)	(3,345)
Loan acquisition costs	(1,444)	(3,480)	(2,733)	(7,945)
Other expenses	(4,975)	(3,468)	(8,659)	(7,553)
(Provision for) Benefit from income taxes	(69)	9,443	1,054	11,901
Net Income (Loss)	2,873	(99,966)	7,493	(69,051)
Non-cash amortization (expense), net	(7,568)	(7,827)	(16,465)	(21,482)
Operating Segments | Residential Mortgage Banking
Segment Reporting Information [Line Items]
Interest income	2,434	13,199	7,944	26,166
Interest expense	(1,700)	(8,297)	(8,566)	(15,233)
Net interest income (expense)	734	4,902	(622)	10,933
Non-interest income (loss)
Mortgage banking activities, net	7,061	(17,787)	10,426	(9,852)
Investment fair value changes, net	0	0	1,076	0
Other income, net	0	0	0	0
Realized gains, net	0	0	0	0
Total non-interest income (loss), net	7,061	(17,787)	11,502	(9,852)
General and administrative expenses	(3,738)	(6,082)	(8,544)	(12,183)
Portfolio management costs	0	0	0	0
Loan acquisition costs	(149)	(881)	(324)	(2,298)
Other expenses	0	74	0	74
(Provision for) Benefit from income taxes	(707)	5,588	(74)	6,595
Net Income (Loss)	3,201	(14,186)	1,938	(6,731)
Non-cash amortization (expense), net	(292)	(760)	(547)	(298)
Operating Segments | Business Purpose Mortgage Banking
Segment Reporting Information [Line Items]
Interest income	4,397	8,586	8,891	13,427
Interest expense	(3,673)	(4,258)	(7,711)	(6,826)
Net interest income (expense)	724	4,328	1,180	6,601
Non-interest income (loss)
Mortgage banking activities, net	9,491	(12,230)	22,797	(3,850)
Investment fair value changes, net	0	0	0	0
Other income, net	1,076	1,054	3,484	1,629
Realized gains, net	0	0	0	0
Total non-interest income (loss), net	10,567	(11,176)	26,281	(2,221)
General and administrative expenses	(11,638)	(11,069)	(25,316)	(21,541)
Portfolio management costs	0	0	0	0
Loan acquisition costs	(1,295)	(2,599)	(2,409)	(5,647)
Other expenses	(3,107)	(3,306)	(6,215)	(6,840)
(Provision for) Benefit from income taxes	1,406	3,169	2,109	6,450
Net Income (Loss)	(3,343)	(20,653)	(4,370)	(23,198)
Non-cash amortization (expense), net	(3,333)	(3,480)	(7,035)	(7,370)
Operating Segments | Investment Portfolio
Segment Reporting Information [Line Items]
Interest income	166,603	144,478	330,263	315,050
Interest expense	(129,855)	(102,589)	(253,307)	(219,171)
Net interest income (expense)	36,748	41,889	76,956	95,879
Non-interest income (loss)
Mortgage banking activities, net	0	0	0	0
Investment fair value changes, net	(1,837)	(98,111)	(2,851)	(103,517)
Other income, net	4,013	6,235	6,181	11,517
Realized gains, net	949	0	832	2,581
Total non-interest income (loss), net	3,125	(91,876)	4,162	(89,419)
General and administrative expenses	(1,241)	(1,274)	(2,650)	(2,829)
Portfolio management costs	(3,087)	(1,767)	(6,597)	(3,345)
Loan acquisition costs	0	0	0	0
Other expenses	(1,868)	(236)	(2,444)	(787)
(Provision for) Benefit from income taxes	(1,465)	686	(1,678)	(1,144)
Net Income (Loss)	32,212	(52,578)	67,749	(1,645)
Non-cash amortization (expense), net	(1,857)	(1,450)	(4,690)	(9,644)
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income	5,547	1,192	10,399	2,212
Interest expense	(17,657)	(11,823)	(35,380)	(22,035)
Net interest income (expense)	(12,110)	(10,631)	(24,981)	(19,823)
Non-interest income (loss)
Mortgage banking activities, net	0	0	0	0
Investment fair value changes, net	(2,759)	10,139	(2,948)	9,425
Other income, net	(931)	(283)	(951)	(157)
Realized gains, net	107	0	222	0
Total non-interest income (loss), net	(3,583)	9,856	(3,677)	9,268
General and administrative expenses	(14,188)	(11,774)	(29,850)	(26,922)
Portfolio management costs	(13)	0	(13)	0
Loan acquisition costs	0	0	0	0
Other expenses	0	0	0	0
(Provision for) Benefit from income taxes	697	0	697	0
Net Income (Loss)	(29,197)	(12,549)	(57,824)	(37,477)
Non-cash amortization (expense), net	$ (2,086)	$ (2,137)	$ (4,193)	$ (4,170)