Use of Special Purpose Entities and Variable Interest Entities (Narrative) (Details) - USD ($) $ in Thousands		3 Months Ended		6 Months Ended
		Jun. 30, 2018	Jun. 30, 2017	Jun. 30, 2018	Jun. 30, 2017	Mar. 31, 2018	Dec. 31, 2017	Mar. 31, 2017	Dec. 31, 2016
Variable Interest Entity [Line Items]
Residential whole loans, at fair value	[1]	$ 1,502,986		$ 1,502,986			$ 1,325,115
Securitized debt		518,655		518,655			363,944
Total residential whole loans		3,400,000		3,400,000			2,200,000
Residential whole loans, at carrying value	[1]	1,906,242		1,906,242			908,516
Real estate owned		192,162	$ 104,443	192,162	$ 104,443	$ 182,940	152,356	$ 98,708	$ 80,503
Other real estate, fair value		214,800		214,800			175,800
Residential whole loans held at carrying value		17,935	8,503	32,264	17,193
Net gain on residential whole loans held at fair value		32,443	$ 16,208	70,941	$ 29,981
Asset-backed Securities, Securitized Loans and Receivables
Variable Interest Entity [Line Items]
Purchased credit impaired loans		199,800		199,800			183,200
Residential whole loans, at fair value		476,200		476,200			289,300
Securitized debt		518,700		518,700			363,900
Other Assets | Asset-backed Securities, Securitized Loans and Receivables
Variable Interest Entity [Line Items]
Real estate owned at fair value		$ 33,400		$ 33,400			$ 5,500

[1] Includes approximately $199.8 million and $183.2 million of Residential whole loans, at carrying value and $476.2 million and $289.3 million of Residential whole loans, at fair value transferred to consolidated variable interest entities (“VIEs”) at June 30, 2018 and December 31, 2017, respectively. Such assets can be used only to settle the obligations of the VIEs.