MBS, CRT Securities and MSR Related Assets (Interest Income) (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Agency MBS
Schedule of Available-for-sale Securities [Line Items]
Coupon interest	$ 96,678	$ 119,966	$ 147,066
Effective yield adjustment	(31,323)	(36,897)	(41,231)
Interest income	65,355	83,069	105,835
Legacy Non-Agency MBS
Schedule of Available-for-sale Securities [Line Items]
Coupon interest	127,645	154,057	183,349
Effective yield adjustment	76,005	78,443	91,003
Interest income	203,650	232,500	274,352
RPL/NPL MBS
Schedule of Available-for-sale Securities [Line Items]
Coupon interest	65,957	98,213	87,429
Effective yield adjustment	1,505	2,108	1,789
Interest income	67,462	100,321	89,218
CRT securities
Schedule of Available-for-sale Securities [Line Items]
Coupon interest	27,706	13,023	5,844
Effective yield adjustment	4,009	1,747	728
Interest income	31,715	14,770	6,572
MSR related assets
Schedule of Available-for-sale Securities [Line Items]
Coupon interest	24,534	2,090	0
Effective yield adjustment	296	10	0
Interest income	$ 24,830	$ 2,100	$ 0