Residential Mortgage Securities and MSR-Related Assets (Unrealized Losses) (Details) $ in Thousands	Sep. 30, 2019 USD ($) security	Dec. 31, 2018 USD ($)
Fair Value
Less than 12 Months	$ 45,973
12 Months or more	859,147
Total	905,120
Unrealized Losses
Less than 12 Months	156
12 Months or more	19,618
Total	$ 19,774
Number of Securities
Less than 12 months (in security) | security	28
12 months or more (in security) | security	392
Available-for-sale securities	$ 4,589,554	$ 6,509,333
Gross unrealized losses	19,774	63,783
Agency MBS
Fair Value
Less than 12 Months	36,775
12 Months or more	824,792
Total	861,567
Unrealized Losses
Less than 12 Months	104
12 Months or more	19,168
Total	$ 19,272
Number of Securities
Less than 12 months (in security) | security	25
12 months or more (in security) | security	382
Available-for-sale securities	$ 1,813,873	2,698,213
Gross unrealized losses	19,272	49,498
Agency MBS | Fannie Mae
Fair Value
Less than 12 Months	36,775
12 Months or more	660,757
Total	697,532
Unrealized Losses
Less than 12 Months	104
12 Months or more	15,178
Total	$ 15,282
Number of Securities
Less than 12 months (in security) | security	25
12 months or more (in security) | security	280
Available-for-sale securities	$ 1,260,016	1,762,032
Gross unrealized losses	15,282	32,321
Agency MBS | Freddie Mac
Fair Value
Less than 12 Months	0
12 Months or more	163,863
Total	163,863
Unrealized Losses
Less than 12 Months	0
12 Months or more	3,990
Total	$ 3,990
Number of Securities
Less than 12 months (in security) | security	0
12 months or more (in security) | security	101
Available-for-sale securities	$ 549,610	931,318
Gross unrealized losses	3,990	17,177
Agency MBS | Ginnie Mae
Fair Value
Less than 12 Months	0
12 Months or more	172
Total	172
Unrealized Losses
Less than 12 Months	0
12 Months or more	0
Total	$ 0
Number of Securities
Less than 12 months (in security) | security	0
12 months or more (in security) | security	1
Available-for-sale securities	$ 4,247	4,863
Gross unrealized losses	0	0
Non-Agency MBS
Fair Value
Less than 12 Months	2,009
12 Months or more	9,389
Total	11,398
Unrealized Losses
Less than 12 Months	11
12 Months or more	250
Total	$ 261
Number of Securities
Less than 12 months (in security) | security	1
12 months or more (in security) | security	3
Available-for-sale securities	$ 2,397,789	3,318,299
Gross unrealized losses	261	8,389
Expected to Recover Par
Fair Value
Less than 12 Months	0
12 Months or more	7,495
Total	7,495
Unrealized Losses
Less than 12 Months	0
12 Months or more	5
Total	$ 5
Number of Securities
Less than 12 months (in security) | security	0
12 months or more (in security) | security	1
Available-for-sale securities	$ 921,705	1,527,700
Gross unrealized losses	5	7,620
Expected to Recover Less Than Par
Fair Value
Less than 12 Months	2,009
12 Months or more	1,894
Total	3,903
Unrealized Losses
Less than 12 Months	11
12 Months or more	245
Total	$ 256
Number of Securities
Less than 12 months (in security) | security	1
12 months or more (in security) | security	2
Available-for-sale securities	$ 1,476,084	1,790,599
Gross unrealized losses	256	769
Total MBS
Fair Value
Less than 12 Months	38,784
12 Months or more	834,181
Total	872,965
Unrealized Losses
Less than 12 Months	115
12 Months or more	19,418
Total	$ 19,533
Number of Securities
Less than 12 months (in security) | security	26
12 months or more (in security) | security	385
Available-for-sale securities	$ 4,211,662	6,016,512
Gross unrealized losses	19,533	57,887
CRT securities
Fair Value
Less than 12 Months	7,189
12 Months or more	24,966
Total	32,155
Unrealized Losses
Less than 12 Months	41
12 Months or more	200
Total	$ 241
Number of Securities
Less than 12 months (in security) | security	2
12 months or more (in security) | security	7
Available-for-sale securities	$ 377,892	492,821
Gross unrealized losses	241	$ 5,896
CRT in Loss Positions Only, Fair Value Option
Number of Securities
Available-for-sale securities	32,200
Gross unrealized losses	$ 241