Other Assets - Real Estate Owned - Activity (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2022 USD ($) property	Dec. 31, 2021 USD ($) property	Dec. 31, 2020 USD ($)
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 156,223	$ 249,699
Adjustments to record at lower of cost or fair value	(4,255)	(4,772)
Transfer from residential whole loans	82,911	72,304	$ 96,766
Purchases and capital improvements, net	978	2,458
Disposals and other	(105,252)	(163,466)
Balance at end of period	$ 130,605	$ 156,223	$ 249,699
Number of properties | property	388	553
Loss recorded on transfer from residential whole loans to real estate owned	$ 1,200	$ 700
Properties sold during period property | property	416	647
Gain on sales of real estate owned	$ 28,700	$ 23,500
Proceeds from sale of real estate	$ 133,800	$ 187,900