Residential Mortgage Securities and MSR-Related Assets (Unrealized Losses) (Details) $ in Thousands	Jun. 30, 2019 USD ($) security	Dec. 31, 2018 USD ($)
Fair Value
Less than 12 Months	$ 152,214
12 Months or more	1,018,360
Total	1,170,574
Unrealized Losses
Less than 12 Months	912
12 Months or more	22,854
Total	$ 23,766
Number of Securities
Less than 12 months (in security) | security	52
12 months or more (in security) | security	390
Available-for-sale securities	$ 5,392,961	$ 6,509,333
Gross unrealized losses	23,766	63,783
Agency MBS
Fair Value
Less than 12 Months	21,017
12 Months or more	924,654
Total	945,671
Unrealized Losses
Less than 12 Months	76
12 Months or more	22,381
Total	$ 22,457
Number of Securities
Less than 12 months (in security) | security	18
12 months or more (in security) | security	384
Available-for-sale securities	$ 2,257,375	2,698,213
Gross unrealized losses	22,457	49,498
Agency MBS | Fannie Mae
Fair Value
Less than 12 Months	21,017
12 Months or more	750,981
Total	771,998
Unrealized Losses
Less than 12 Months	76
12 Months or more	17,467
Total	$ 17,543
Number of Securities
Less than 12 months (in security) | security	18
12 months or more (in security) | security	283
Available-for-sale securities	$ 1,463,418	1,762,032
Gross unrealized losses	17,543	32,321
Agency MBS | Freddie Mac
Fair Value
Less than 12 Months	0
12 Months or more	173,673
Total	173,673
Unrealized Losses
Less than 12 Months	0
12 Months or more	4,914
Total	$ 4,914
Number of Securities
Less than 12 months (in security) | security	0
12 months or more (in security) | security	101
Available-for-sale securities	$ 789,470	931,318
Gross unrealized losses	4,914	17,177
Non-Agency MBS
Fair Value
Less than 12 Months	2,391
12 Months or more	93,706
Total	96,097
Unrealized Losses
Less than 12 Months	10
12 Months or more	473
Total	$ 483
Number of Securities
Less than 12 months (in security) | security	2
12 months or more (in security) | security	6
Available-for-sale securities	$ 2,728,270	3,318,299
Gross unrealized losses	483	8,389
Non-Agency MBS | Expected to Recover Par
Fair Value
Less than 12 Months	0
12 Months or more	90,058
Total	90,058
Unrealized Losses
Less than 12 Months	0
12 Months or more	174
Total	$ 174
Number of Securities
Less than 12 months (in security) | security	0
12 months or more (in security) | security	4
Available-for-sale securities	$ 1,153,607	1,527,700
Gross unrealized losses	174	7,620
Non-Agency MBS | Expected to Recover Less Than Par
Fair Value
Less than 12 Months	2,391
12 Months or more	3,648
Total	6,039
Unrealized Losses
Less than 12 Months	10
12 Months or more	299
Total	$ 309
Number of Securities
Less than 12 months (in security) | security	2
12 months or more (in security) | security	2
Available-for-sale securities	$ 1,574,663	1,790,599
Gross unrealized losses	309	769
Total MBS
Fair Value
Less than 12 Months	23,408
12 Months or more	1,018,360
Total	1,041,768
Unrealized Losses
Less than 12 Months	86
12 Months or more	22,854
Total	$ 22,940
Number of Securities
Less than 12 months (in security) | security	20
12 months or more (in security) | security	390
Available-for-sale securities	$ 4,985,645	6,016,512
Gross unrealized losses	22,940	57,887
CRT securities
Fair Value
Less than 12 Months	128,806
12 Months or more	0
Total	128,806
Unrealized Losses
Less than 12 Months	826
12 Months or more	0
Total	$ 826
Number of Securities
Less than 12 months (in security) | security	32
12 months or more (in security) | security	0
Available-for-sale securities	$ 407,316	492,821
Gross unrealized losses	826	$ 5,896
CRT in Loss Positions Only, Fair Value Option
Number of Securities
Available-for-sale securities	128,800
Gross unrealized losses	$ 826