MBS, CRT Securities and MSR Related Assets (Narrative) (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Schedule of Available-for-sale Securities [Line Items]
Minimum term of fixed rate mortgages underlying MBS, years									15 years
Available-for-sale securities, continuous unrealized loss position, accumulated loss	$ 43,613								$ 43,613
Reclassification adjustment for other-than-temporary impairments included in net income	0	$ 0	$ 618	$ 414	$ 0	$ 485	$ 0	$ 0	1,032	$ 485	$ 705
Agency MBS
Schedule of Available-for-sale Securities [Line Items]
Available-for-sale securities, continuous unrealized loss position, accumulated loss	43,137								$ 43,137
RPL/NPL MBS
Schedule of Available-for-sale Securities [Line Items]
Debt instrument, basis spread on variable rate									3.00%
Debt instrument, coupon step-up period									36 months
Non-Agency MBS
Schedule of Available-for-sale Securities [Line Items]
Available-for-sale securities, continuous unrealized loss position, accumulated loss	$ 453								$ 453