Other Assets (Real Estate Owned - Activity) (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2020 USD ($) property	Dec. 31, 2019 USD ($) property	Dec. 31, 2018 USD ($)
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 411,659	$ 249,413
Adjustments to record at lower of cost or fair value	(12,570)	(14,884)
Transfer from residential whole loans	96,766	257,701	$ 215,038
Purchases and capital improvements, net	10,198	20,746
Disposals	(256,354)	(101,317)
Balance at end of period	$ 249,699	$ 411,659	$ 249,413
Number of properties | property	946	1,652
Gain recorded on transfer from residential whole loans to real estate owned	$ 5,100	$ 19,800
Properties sold during period property | property	1,086	571
Gain on sales of real estate owned	$ 15,100	$ 7,400
Proceeds from sale of real estate	$ 271,400	$ 109,200