Residential Morgage Securities and MSR Related Assets (Impact of AFS Securities on AOCI) (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2018	Dec. 31, 2017	Dec. 31, 2016
AOCI from AFS Securities:
Unrealized gain on AFS securities at beginning of period	$ 620,648	$ 620,403	$ 585,250
Reclassification adjustment for MBS sales included in net income	(51,580)	(38,707)	(36,922)
Reclassification adjustment for OTTI included in net income	(1,259)	(1,032)	(485)
Change in AOCI from AFS securities	(203,481)	245	35,153
Balance at end of period	417,167	620,648	620,403
Agency MBS
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	(17,891)	(39,158)	(9,322)
Non-Agency MBS
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	(131,939)	78,337	81,882
Unrealized (loss)/gain on MSR term notes, net
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	$ (812)	$ 805	$ 0