Other Assets - Real Estate Owned - Activity (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2021 USD ($) property	Dec. 31, 2020 USD ($) property	Dec. 31, 2019 USD ($)
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 249,699	$ 411,659
Adjustments to record at lower of cost or fair value	(4,772)	(12,570)
Transfer from residential whole loans	72,304	96,766	$ 257,701
Purchases and capital improvements, net	2,458	10,198
Disposals and other	(163,466)	(256,354)
Balance at end of period	$ 156,223	$ 249,699	$ 411,659
Number of properties | property	553	946
Gain recorded on transfer from residential whole loans to real estate owned	$ (700)	$ 5,100
Properties sold during period property | property	647	1,086
Gain on sales of real estate owned	$ 23,500	$ 15,100
Proceeds from sale of real estate	$ 187,900	$ 274,100