Residential Mortgage Securities and MSR-Related Assets (Unrealized Losses) (Details) $ in Thousands	Sep. 30, 2020 USD ($) security	Dec. 31, 2019 USD ($)
Fair Value
Less than 12 Months	$ 104,480
12 Months or more	0
Total	104,480
Unrealized Losses
Less than 12 Months	6,092
12 Months or more	0
Total	$ 6,092
Number of Securities
Less than 12 months (in security) | security	13
12 months or more (in security) | security	0
Fair  Value	$ 152,765	$ 3,983,519
Gross unrealized losses	6,092	18,818
Non-Agency MBS
Unrealized Losses
Total	3,000
Number of Securities
Fair  Value	56,430	2,063,529
Gross unrealized losses	2,965	18
Total MBS
Fair Value
Less than 12 Months	44,927
12 Months or more	0
Total	44,927
Unrealized Losses
Less than 12 Months	2,965
12 Months or more	0
Total	$ 2,965
Number of Securities
Less than 12 months (in security) | security	5
12 months or more (in security) | security	0
Fair  Value		3,728,111
Gross unrealized losses		18,727
CRT securities
Fair Value
Less than 12 Months	$ 59,553
12 Months or more	0
Total	59,553
Unrealized Losses
Less than 12 Months	3,127
12 Months or more	0
Total	$ 3,127
Number of Securities
Less than 12 months (in security) | security	8
12 months or more (in security) | security	0
Fair  Value	$ 96,335	255,408
Gross unrealized losses	3,127	$ 91
CRT in Loss Positions Only, Fair Value Option
Number of Securities
Fair  Value	59,600
Gross unrealized losses	$ 3,100