Other Assets - Real Estate Owned - Activity (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2022 USD ($) property	Mar. 31, 2021 USD ($) property	Dec. 31, 2021 property
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 156,223	$ 249,699
Adjustments to record at lower of cost or fair value	(448)	(874)
Transfer from residential whole loans	22,079	20,068
Purchases and capital improvements, net	353	217
Disposals and other	(32,639)	(48,717)
Balance at end of period	$ 145,568	$ 220,393
Number of properties | property	492	835	553
Gain recorded on transfer from residential whole loans to real estate owned	$ (100)	$ 1,100
Properties sold during period | property	135	177
Proceeds from sale of real estate	$ 41,500	$ 50,600
Gain on sales of real estate owned	$ 8,700	$ 2,200