MBS, CRT Securities and MSR Related Assets (Impact of AFS Securities on AOCI) (Details) - USD ($)	3 Months Ended		9 Months Ended
	Sep. 30, 2018	Sep. 30, 2017	Sep. 30, 2018	Sep. 30, 2017
AOCI from AFS Securities:
Unrealized gain on AFS securities at beginning of period	$ 548,551,000	$ 668,223,000	$ 620,648,000	$ 620,403,000
Reclassification adjustment for MBS sales included in net income	(9,455,000)	(14,935,000)	(25,580,000)	(30,283,000)
Reclassification adjustment for OTTI included in net income	0	0	0	(1,032,000)
Change in AOCI from AFS securities	(43,733,000)	(7,947,000)	(115,830,000)	39,873,000
Balance at end of period	504,818,000	660,276,000	504,818,000	660,276,000
Agency MBS
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	(9,177,000)	(3,032,000)	(27,507,000)	(22,241,000)
Non-Agency MBS
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	$ (25,101,000)	$ 10,020,000	$ (62,743,000)	$ 93,429,000