Fair Value of Financial Instruments (Fair Value Methodology) (Details) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2018 USD ($)	Dec. 31, 2017 USD ($)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	$ 1,468,480	$ 951,811
Purchases excluded from level 2 fair value	34,500	373,300
Discounted cash flow
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	673,765	358,871
Liquidation model
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	794,715	592,940
Simple average amount	$ 384	$ 336
Residential whole loans | Level 3 | Liquidation model | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Annual change in home prices	(0.60%)	(8.00%)
Liquidation timeline (in years)	1 month 6 days	1 month 6 days
Current value of underlying properties	$ 1	$ 0
Residential whole loans | Level 3 | Liquidation model | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Annual change in home prices	11.20%	8.80%
Liquidation timeline (in years)	4 years 6 months	4 years 6 months
Current value of underlying properties	$ 12,400	$ 9,900
Discount rate | Residential whole loans | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.045	0.045
Discount rate | Residential whole loans | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.082	0.130
Discount rate | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.055	0.055
Discount rate | Residential whole loans | Level 3 | Liquidation model | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.061	0.061
Discount rate | Residential whole loans | Level 3 | Liquidation model | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.500	0.500
Discount rate | Residential whole loans | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.082	0.080
Prepayment rate | Residential whole loans | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.009	0.012
Prepayment rate | Residential whole loans | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.135	0.151
Prepayment rate | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.039	0.041
Prepayment rate | Residential whole loans | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.031	0.025
Default rate | Residential whole loans | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Default rate | Residential whole loans | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.208	0.065
Default rate | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.025	0.029
Default rate | Residential whole loans | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Liquidation timeline (in years)	1 year 9 months 18 days	1 year 7 months 6 days
Loss severity | Residential whole loans | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Loss severity | Residential whole loans | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	1.000	1.000
Loss severity | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.130	0.138
Loss severity | Residential whole loans | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Current value of underlying properties	$ 829	$ 772