Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2022	Jun. 30, 2021	Jun. 30, 2022	Jun. 30, 2021
Segment Reporting Information [Line Items]
Interest income	$ 167,455	$ 138,695	$ 356,855	$ 267,000
Interest expense	(126,967)	(108,065)	(263,265)	(210,617)
Net Interest Income	40,488	30,630	93,590	56,383
Non-interest (loss) income
Mortgage banking activities, net	(30,017)	54,419	(13,702)	137,026
Investment fair value changes, net	(87,972)	49,480	(94,092)	94,567
Other income, net	7,006	2,126	12,989	5,969
Realized gains, net	0	8,384	2,581	11,100
Total non-interest (loss) income, net	(110,983)	114,409	(92,224)	248,662
General and administrative expenses	(31,966)	(40,594)	(66,820)	(84,145)
Loan acquisition costs	(3,480)	(3,748)	(7,945)	(7,307)
Other expenses	(3,468)	(3,985)	(7,553)	(8,081)
Benefit from (provision for) income taxes	9,443	(6,687)	11,901	(18,230)
Net (Loss) Income	(99,966)	90,025	(69,051)	187,282
Non-cash amortization (expense) income, net	(7,827)	(6,015)	(21,482)	(12,245)
Operating Segments | Residential Mortgage Banking
Segment Reporting Information [Line Items]
Interest income	13,199	13,246	26,166	20,825
Interest expense	(8,297)	(7,618)	(15,233)	(12,365)
Net Interest Income	4,902	5,628	10,933	8,460
Non-interest (loss) income
Mortgage banking activities, net	(17,787)	21,265	(9,852)	82,700
Investment fair value changes, net	0	0	0	0
Other income, net	0	0	0	0
Realized gains, net	0	0	0	0
Total non-interest (loss) income, net	(17,787)	21,265	(9,852)	82,700
General and administrative expenses	(6,082)	(6,898)	(12,183)	(19,587)
Loan acquisition costs	(881)	(1,887)	(2,298)	(3,291)
Other expenses	74	0	74	(6)
Benefit from (provision for) income taxes	5,588	(3,725)	6,595	(13,211)
Net (Loss) Income	(14,186)	14,383	(6,731)	55,065
Non-cash amortization (expense) income, net	(760)	(13)	(298)	(16)
Operating Segments | Business Purpose Mortgage Banking
Segment Reporting Information [Line Items]
Interest income	8,586	3,026	13,427	5,881
Interest expense	(4,258)	(1,521)	(6,826)	(3,120)
Net Interest Income	4,328	1,505	6,601	2,761
Non-interest (loss) income
Mortgage banking activities, net	(12,230)	33,154	(3,850)	54,326
Investment fair value changes, net	0	0	0	0
Other income, net	1,054	156	1,629	278
Realized gains, net	0	0	0	0
Total non-interest (loss) income, net	(11,176)	33,310	(2,221)	54,604
General and administrative expenses	(11,069)	(12,356)	(21,541)	(22,550)
Loan acquisition costs	(2,599)	(1,861)	(5,647)	(3,353)
Other expenses	(3,306)	(3,873)	(6,840)	(7,650)
Benefit from (provision for) income taxes	3,169	(2,182)	6,450	(3,503)
Net (Loss) Income	(20,653)	14,543	(23,198)	20,309
Non-cash amortization (expense) income, net	(3,480)	(4,054)	(7,370)	(8,152)
Operating Segments | Investment Portfolio
Segment Reporting Information [Line Items]
Interest income	144,478	121,248	315,050	237,757
Interest expense	(102,589)	(88,693)	(219,171)	(174,638)
Net Interest Income	41,889	32,555	95,879	63,119
Non-interest (loss) income
Mortgage banking activities, net	0	0	0	0
Investment fair value changes, net	(98,111)	49,727	(103,517)	95,488
Other income, net	6,235	1,705	11,517	5,279
Realized gains, net	0	8,384	2,581	11,100
Total non-interest (loss) income, net	(91,876)	59,816	(89,419)	111,867
General and administrative expenses	(3,041)	(3,157)	(6,174)	(6,321)
Loan acquisition costs	0	0	0	(659)
Other expenses	(236)	(112)	(787)	(442)
Benefit from (provision for) income taxes	686	(780)	(1,144)	(1,516)
Net (Loss) Income	(52,578)	88,322	(1,645)	166,048
Non-cash amortization (expense) income, net	(1,450)	7	(9,644)	(227)
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income	1,192	1,175	2,212	2,537
Interest expense	(11,823)	(10,233)	(22,035)	(20,494)
Net Interest Income	(10,631)	(9,058)	(19,823)	(17,957)
Non-interest (loss) income
Mortgage banking activities, net	0	0	0	0
Investment fair value changes, net	10,139	(247)	9,425	(921)
Other income, net	(283)	265	(157)	412
Realized gains, net	0	0	0	0
Total non-interest (loss) income, net	9,856	18	9,268	(509)
General and administrative expenses	(11,774)	(18,183)	(26,922)	(35,687)
Loan acquisition costs	0	0	0	(4)
Other expenses	0	0	0	17
Benefit from (provision for) income taxes	0	0	0	0
Net (Loss) Income	(12,549)	(27,223)	(37,477)	(54,140)
Non-cash amortization (expense) income, net	$ (2,137)	$ (1,955)	$ (4,170)	$ (3,850)