MBS, CRT Securities and MSR Related Assets (Unrealized Losses) (Details) $ in Thousands	Jun. 30, 2018 USD ($) security
Fair Value
Less than 12 Months	$ 706,393
12 Months or more	1,196,589
Total	1,902,982
Unrealized Losses
Less than 12 Months	5,684
12 Months or more	50,257
Total	$ 55,941
Number of Securities
Less than 12 months (in security) | security	163
12 months or more (in security) | security	341
Agency MBS
Fair Value
Less than 12 Months	$ 394,294
12 Months or more	1,188,415
Total	1,582,709
Unrealized Losses
Less than 12 Months	4,428
12 Months or more	50,134
Total	$ 54,562
Number of Securities
Less than 12 months (in security) | security	140
12 months or more (in security) | security	333
Agency MBS | Fannie Mae
Fair Value
Less than 12 Months	$ 304,882
12 Months or more	888,232
Total	1,193,114
Unrealized Losses
Less than 12 Months	3,195
12 Months or more	32,371
Total	$ 35,566
Number of Securities
Less than 12 months (in security) | security	97
12 months or more (in security) | security	237
Agency MBS | Freddie Mac
Fair Value
Less than 12 Months	$ 88,740
12 Months or more	300,183
Total	388,923
Unrealized Losses
Less than 12 Months	1,232
12 Months or more	17,763
Total	$ 18,995
Number of Securities
Less than 12 months (in security) | security	40
12 months or more (in security) | security	96
Agency MBS | Government National Mortgage Association Certificates and Obligations (GNMA) [Member]
Fair Value
Less than 12 Months	$ 672
12 Months or more	0
Total	672
Unrealized Losses
Less than 12 Months	1
12 Months or more	0
Total	$ 1
Number of Securities
Less than 12 months (in security) | security	3
12 months or more (in security) | security	0
Non-Agency MBS
Fair Value
Less than 12 Months	$ 294,035
12 Months or more	8,174
Total	302,209
Unrealized Losses
Less than 12 Months	1,163
12 Months or more	123
Total	$ 1,286
Number of Securities
Less than 12 months (in security) | security	18
12 months or more (in security) | security	8
Non-Agency MBS | Expected to Recover Par
Fair Value
Less than 12 Months	$ 278,307
12 Months or more	8,174
Total	286,481
Unrealized Losses
Less than 12 Months	971
12 Months or more	123
Total	$ 1,094
Number of Securities
Less than 12 months (in security) | security	14
12 months or more (in security) | security	8
Non-Agency MBS | Expected to Recover Less Than Par
Fair Value
Less than 12 Months	$ 15,728
12 Months or more	0
Total	15,728
Unrealized Losses
Less than 12 Months	192
12 Months or more	0
Total	$ 192
Number of Securities
Less than 12 months (in security) | security	4
12 months or more (in security) | security	0
Total MBS
Fair Value
Less than 12 Months	$ 688,329
12 Months or more	1,196,589
Total	1,884,918
Unrealized Losses
Less than 12 Months	5,591
12 Months or more	50,257
Total	$ 55,848
Number of Securities
Less than 12 months (in security) | security	158
12 months or more (in security) | security	341
CRT securities
Fair Value
Less than 12 Months	$ 18,064
12 Months or more	0
Total	18,064
Unrealized Losses
Less than 12 Months	93
12 Months or more	0
Total	$ 93
Number of Securities
Less than 12 months (in security) | security	5
12 months or more (in security) | security	0