Other Assets - Schedule of Other Assets (Details) - USD ($) $ in Thousands		Sep. 30, 2023	Jun. 30, 2023	Dec. 31, 2022	Sep. 30, 2022	Jun. 30, 2022	Dec. 31, 2021
Deferred Costs, Capitalized, Prepaid, and Other Assets Disclosure [Abstract]
Receivable for sale of unsettled residential whole loans		$ 0		$ 275,656
REO		113,090	$ 119,996	130,605	$ 132,663	$ 135,847	$ 156,223
Goodwill		61,076		61,076
Intangibles, net		8,800		12,200
Capital contributions made to loan origination partners		20,034		28,308
Commercial loans		76,546		61,510
Interest receivable		89,554		68,704
Other loan related receivables		30,649		23,463
Lease Right-of-Use Asset		37,762		39,459
Other		65,576		65,240
Total Other Assets	[1]	$ 503,087		$ 766,221
Operating Lease, Right-of-Use Asset, Statement of Financial Position [Extensible Enumeration]		Total Other Assets		Total Other Assets
Accumulated amortization		$ 19,200		$ 15,800
Borrowing rate (as a percent)		7.50%

[1] See Note 6 for information regarding the Company’s pledged assets.