Other Assets (Real Estate Owned - Activity) (Details) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2020 USD ($) property	Jun. 30, 2019 USD ($) property	Jun. 30, 2020 USD ($) property	Jun. 30, 2019 USD ($) property	Dec. 31, 2019 property
Real Estate Owned [Roll Forward]
Balance at beginning of period	$ 411,473	$ 290,587	$ 411,659	$ 249,413
Adjustments to record at lower of cost or fair value	(7,138)	(1,315)	(11,889)	(5,388)
Transfer from residential whole loans	8,526	66,483	59,219	131,644
Purchases and capital improvements, net	3,192	5,274	8,798	11,197
Disposals	(67,537)	(26,960)	(119,271)	(52,797)
Balance at end of period	$ 348,516	$ 334,069	$ 348,516	$ 334,069
Number of properties | property	1,352	1,362	1,352	1,362	1,652
Gain recorded on transfer from residential whole loans to real estate owned	$ 200	$ 6,500	$ 3,200	$ 11,300
Properties sold during period | property	296	152	545	289
Proceeds from sale of real estate	$ 70,500	$ 29,200	$ 125,300	$ 57,000
Gain on sales of real estate owned	$ 3,000	$ 2,300	$ 6,104	$ 3,692