Residential Mortgage Securities and MSR-Related Assets (Impact of AFS Securities on AOCI) (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2019	Jun. 30, 2018	Jun. 30, 2019	Jun. 30, 2018
AOCI from AFS Securities:
Unrealized gain on AFS securities at beginning of period	$ 422,261	$ 574,485	$ 417,167	$ 620,648
Reclassification adjustment for MBS sales included in net income	(6,371)	(5,178)	(21,576)	(15,458)
Change in AOCI from AFS securities	17,637	(25,934)	22,731	(72,097)
Balance at end of period	439,898	548,551	439,898	548,551
Agency MBS
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	13,555	(9,641)	21,880	(18,331)
Non-Agency MBS
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	7,598	(11,115)	19,060	(38,544)
MSR-related assets
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net	$ 2,855	$ 0	$ 3,367	$ 236