MBS, CRT Securities and MSR Related Assets (Unrealized Losses) (Details)	Dec. 31, 2017 USD ($) security	Dec. 31, 2016 USD ($)
Fair Value
Less than 12 Months	$ 536,893,000
12 Months or more	1,292,244,000
Total	1,829,137,000
Available-for-sale Securities, Accumulated Gross Unrealized Loss, before Tax	43,613,000	$ 36,398,000
Unrealized Losses
Less than 12 Months	3,837,000
12 Months or more	39,776,000
Total	$ 43,613,000
Number of Securities
Less than 12 months (in security) | security	138
12 months or more (in security) | security	314
Agency MBS
Fair Value
Less than 12 Months	$ 449,996,000
12 Months or more	1,279,713,000
Total	1,729,709,000
Available-for-sale Securities, Accumulated Gross Unrealized Loss, before Tax	43,137,000	31,193,000
Unrealized Losses
Less than 12 Months	3,600,000
12 Months or more	39,537,000
Total	$ 43,137,000
Number of Securities
Less than 12 months (in security) | security	128
12 months or more (in security) | security	305
Agency MBS | Fannie Mae
Fair Value
Less than 12 Months	$ 385,839,000
12 Months or more	871,828,000
Total	1,257,667,000
Available-for-sale Securities, Accumulated Gross Unrealized Loss, before Tax	28,341,000	19,308,000
Unrealized Losses
Less than 12 Months	3,128,000
12 Months or more	25,213,000
Total	$ 28,341,000
Number of Securities
Less than 12 months (in security) | security	104
12 months or more (in security) | security	200
Agency MBS | Freddie Mac
Fair Value
Less than 12 Months	$ 63,924,000
12 Months or more	407,885,000
Total	471,809,000
Available-for-sale Securities, Accumulated Gross Unrealized Loss, before Tax	14,796,000	11,885,000
Unrealized Losses
Less than 12 Months	472,000
12 Months or more	14,324,000
Total	$ 14,796,000
Number of Securities
Less than 12 months (in security) | security	23
12 months or more (in security) | security	105
Agency MBS | Ginnie Mae
Fair Value
Less than 12 Months	$ 233,000
12 Months or more	0
Total	233,000
Available-for-sale Securities, Accumulated Gross Unrealized Loss, before Tax	0	0
Unrealized Losses
Less than 12 Months	0
12 Months or more	0
Total	$ 0
Number of Securities
Less than 12 months (in security) | security	1
12 months or more (in security) | security	0
Non-Agency MBS
Fair Value
Less than 12 Months	$ 70,631,000
12 Months or more	12,531,000
Total	83,162,000
Available-for-sale Securities, Accumulated Gross Unrealized Loss, before Tax	453,000	5,202,000
Unrealized Losses
Less than 12 Months	214,000
12 Months or more	239,000
Total	$ 453,000
Number of Securities
Less than 12 months (in security) | security	6
12 months or more (in security) | security	9
Non-Agency MBS | Expected to Recover Par
Fair Value
Less than 12 Months	$ 64,394,000
12 Months or more	12,531,000
Total	76,925,000
Available-for-sale Securities, Accumulated Gross Unrealized Loss, before Tax	434,000	2,368,000
Unrealized Losses
Less than 12 Months	195,000
12 Months or more	239,000
Total	$ 434,000
Number of Securities
Less than 12 months (in security) | security	2
12 months or more (in security) | security	9
Non-Agency MBS | Expected to Recover Less Than Par
Fair Value
Less than 12 Months	$ 6,237,000
12 Months or more	0
Total	6,237,000
Available-for-sale Securities, Accumulated Gross Unrealized Loss, before Tax	19,000	2,834,000
Unrealized Losses
Less than 12 Months	19,000
12 Months or more	0
Total	$ 19,000
Number of Securities
Less than 12 months (in security) | security	4
12 months or more (in security) | security	0
Total MBS
Fair Value
Less than 12 Months	$ 520,627,000
12 Months or more	1,292,244,000
Total	1,812,871,000
Available-for-sale Securities, Accumulated Gross Unrealized Loss, before Tax	43,590,000	36,395,000
Unrealized Losses
Less than 12 Months	3,814,000
12 Months or more	39,776,000
Total	$ 43,590,000
Number of Securities
Less than 12 months (in security) | security	134
12 months or more (in security) | security	314
CRT securities
Fair Value
Less than 12 Months	$ 16,266,000
12 Months or more	0
Total	16,266,000
Available-for-sale Securities, Accumulated Gross Unrealized Loss, before Tax	23,000	$ 3,000
Unrealized Losses
Less than 12 Months	23,000
12 Months or more	0
Total	$ 23,000
Number of Securities
Less than 12 months (in security) | security	4
12 months or more (in security) | security	0
Fair value option elected, debt securities | CRT securities
Fair Value
Total	$ 16,300,000
Available-for-sale Securities, Accumulated Gross Unrealized Loss, before Tax	$ 23,000