Other Assets - Real Estate Owned - Activity (Details)	3 Months Ended		9 Months Ended
	Sep. 30, 2022 USD ($) property	Sep. 30, 2021 USD ($) property	Sep. 30, 2022 USD ($) property	Sep. 30, 2021 USD ($) property	Dec. 31, 2021 property
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 135,847,000	$ 204,762,000	$ 156,223,000	$ 249,699,000
Adjustments to record at lower of cost or fair value	(1,655,000)	(2,448,000)	(2,426,000)	(3,390,000)
Transfer from residential whole loans	16,658,000	11,673,000	61,875,000	50,043,000
Purchases and capital improvements, net	215,000	1,584,000	845,000	2,207,000
Disposals and other	(18,402,000)	(36,769,000)	(83,854,000)	(119,757,000)
Balance at end of period	$ 132,663,000	$ 178,802,000	$ 132,663,000	$ 178,802,000
Number of properties | property	412	674	412	674	553
Gain (loss) recorded on transfer from residential whole loans to real estate owned	$ (300,000)	$ 700,000	$ (300,000)	$ 400,000
Properties sold during period | property	74	151	333	470
Proceeds from sale of real estate	$ 23,800,000	$ 45,400,000	$ 105,000,000	$ 134,000,000
Gain on sales of real estate owned	$ 5,300,000	$ 7,300,000	$ 21,300,000	$ 13,400,000