Residential Mortgage Securities and MSR-Related Assets (Purchase Discounts) (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2019	Jun. 30, 2018	Jun. 30, 2019	Jun. 30, 2018
Discount Designated as Credit Reserve and OTTI | Non-Agency MBS
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period	$ (501,619)	$ (572,580)	$ (516,116)	$ (593,227)
Impact of RMBS Issuer Settlement	0	0	0	0
Accretion of discount	0	0	0	0
Realized credit losses	9,917	10,954	17,420	19,401
Purchases	(624)	0	(624)	(535)
Sales	8,171	0	11,363	5,592
Transfers/release of credit reserve	4,589	8,030	8,391	15,173
Balance at end of period	(479,566)	(553,596)	(479,566)	(553,596)
Accretable Discount | Non-Agency MBS
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period	(130,147)	(199,659)	(155,025)	(215,325)
Impact of RMBS Issuer Settlement	(833)	(12,089)	(1,688)	(12,089)
Accretion of discount	14,551	17,530	27,858	34,746
Realized credit losses	0	0	0	0
Purchases	409	0	291	488
Sales	2,856	0	19,202	5,105
Transfers/release of credit reserve	(4,589)	(8,030)	(8,391)	(15,173)
Balance at end of period	(117,753)	(202,248)	(117,753)	(202,248)
Jp Morgan Chase
Changes in the components of the purchase discount on Non-Agency MBS
Proceeds from legal settlements	$ 833	$ 12,100	$ 1,700	$ 12,100