Other Assets - Real Estate Owned - Activity (Details)	3 Months Ended		6 Months Ended
	Jun. 30, 2022 USD ($) property	Jun. 30, 2021 USD ($) property	Jun. 30, 2022 USD ($) property	Jun. 30, 2021 USD ($) property	Dec. 31, 2021 property
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 145,568,000	$ 220,393,000	$ 156,223,000	$ 249,699,000
Adjustments to record at lower of cost or fair value	(324,000)	(68,000)	(772,000)	(943,000)
Transfer from residential whole loans	23,138,000	18,302,000	45,217,000	38,370,000
Purchases and capital improvements, net	277,000	406,000	629,000	624,000
Disposals and other	(32,812,000)	(34,271,000)	(65,450,000)	(82,988,000)
Balance at end of period	$ 135,847,000	$ 204,762,000	$ 135,847,000	$ 204,762,000
Number of properties | property	438	766	438	766	553
Gain (loss) recorded on transfer from residential whole loans to real estate owned	$ (9,000)	$ 22,000	$ 400	$ 1,100,000
Properties sold during period | property	124	139	259	319
Proceeds from sale of real estate	$ 39,600,000	$ 38,000,000	$ 81,200,000	$ 88,600,000
Gain on sales of real estate owned	$ 7,300,000	$ 3,900,000	$ 16,000,000	$ 6,100,000