Residential Mortgage Securities and MSR-Related Assets (Impact of AFS Securities on AOCI) (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2019	Mar. 31, 2018
AOCI from AFS Securities:
Unrealized gain on AFS securities at beginning of period	$ 417,167	$ 620,648
Reclassification adjustment for MBS sales included in net income	(17,009)	(8,623)
Change in AOCI from AFS securities	5,094	(46,163)
Balance at end of period	422,261	574,485
Agency MBS
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	9,315	(10,052)
Non-Agency MBS
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	12,276	(27,724)
MSR-related assets
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	$ 512	$ 236