Fair Value of Financial Instruments (Fair Value Methodology) (Details) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2019 USD ($)	Dec. 31, 2018 USD ($)
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	$ 1,435,367	$ 1,383,502
Purchases excluded from level 2 fair value	90,400	282,500
Discounted cash flow
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	787,259	700,250
Liquidation model
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Residential whole loans, at fair value	648,108	683,252
Simple average amount	$ 363	$ 400
Residential whole loans, at fair value | Level 3 | Liquidation model | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Annual change in home prices	0.90%	(0.50%)
Liquidation timeline (in years)	1 month 6 days	1 month 6 days
Current value of underlying properties	$ 4	$ 2
Residential whole loans, at fair value | Level 3 | Liquidation model | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Annual change in home prices	8.30%	12.20%
Liquidation timeline (in years)	4 years 6 months	4 years 6 months
Current value of underlying properties	$ 4,500	$ 7,950
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.041	0.045
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.080	0.080
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.047	0.052
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.057	0.061
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.500	0.500
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.082	0.080
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.007	0.0090
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.180	0.159
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.044	0.048
Prepayment rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Annual change in home prices	3.50%	3.50%
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.235	0.241
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.046	0.041
Default rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Liquidation timeline (in years)	1 year 9 months 18 days	1 year 9 months 18 days
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	1.000	1.000
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.130	0.129
Loss severity | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Current value of underlying properties	$ 689	$ 802