Fair Value of Financial Instruments - Level 3 Assets (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2017	Sep. 30, 2016	Sep. 30, 2017	Sep. 30, 2016	Jun. 30, 2017
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans held at fair value excluded, purchase transaction not occurred	$ 120,400	$ 92,800	$ 120,400	$ 92,800	$ 92,700
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Changes in unrealized gain/losses	563	0	563	0
Excluded amount, unsettled purchases, residential whole loans, at carrying value		111,200		111,200	$ 97,900
Recurring basis | Level 3 | Residential Whole Loans, at Fair Value (1)
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	744,072	684,582	814,682	623,276
Purchases and capitalized advances	284,930	50,071	295,094	169,830
Changes in fair value recorded in Net gain on residential whole loans held at fair value	5,289	10,913	12,499	25,529
Collection of principal, net of liquidation gains/losses	(17,670)	(18,119)	(53,366)	(48,909)
Repurchases	(257)	0	(1,013)	0
Transfers	(33,214)	(22,985)	(84,746)	(65,264)
Balance at end of period	983,150	704,462	983,150	704,462
Recurring basis | Level 3 | Term Notes Backed by MSR Related Collateral
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	273,961	0	0	0
Purchases and capitalized advances	161,000	0	311,000	0
Collection of principal, net of liquidation gains/losses	(123,961)	0	(140,980)	0
Transfers	0	0	140,980	0
Balance at end of period	$ 311,563	$ 0	$ 311,563	$ 0