Other Assets (Real Estate Owned - Activity) (Details)	3 Months Ended		9 Months Ended
	Sep. 30, 2021 USD ($) property	Sep. 30, 2020 USD ($) property	Sep. 30, 2021 USD ($) property loan	Sep. 30, 2020 USD ($) property	Dec. 31, 2020 property
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 204,762,000	$ 348,516,000	$ 249,699,000	$ 411,659,000
Adjustments to record at lower of cost or fair value	2,448,000	(93,000)	3,390,000	11,796,000
Transfer from residential whole loans	11,673,000	15,672,000	50,043,000	74,891,000
Purchases and capital improvements, net	1,584,000	771,000	2,207,000	10,072,000
Disposals	(36,769,000)	(66,185,000)	(119,757,000)	(185,959,000)
Balance at end of period	$ 178,802,000	$ 298,867,000	$ 178,802,000	$ 298,867,000
Number of properties | property	674	1,131	674	1,131	946
Gain recorded on transfer from residential whole loans to real estate owned	$ 700,000	$ 834,000	$ (400,000)	$ 4,100,000
Properties sold during period	151	267	470	812
Proceeds from sale of real estate	$ 45,400,000	$ 69,900,000	$ 134,000,000	$ 195,200,000
Gain on sales of real estate owned	$ 7,300,000	$ 3,900,000	$ 13,400,000	$ 10,000,000