Fair Value of Financial Instruments (Fair Value Methodology) (Details) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2021 USD ($)	Dec. 31, 2020 USD ($)
Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	$ 1,168,185	$ 1,216,637
Purchases excluded from level 2 fair value	266	265
Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	835,177
Purchases excluded from level 2 fair value	131,300
Discounted cash flow | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	773,449	789,576
Discounted cash flow | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	832,928
Liquidation model | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	394,736	427,061
Simple average amount	410	$ 380
Liquidation model | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	$ 2,249
Residential whole loans | Level 3 | Liquidation model | Minimum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	4.30%	1.80%
Liquidation timeline (in years)	8 months 12 days	9 months 18 days
Fair Value Inputs, Value of Underlying Property	$ 5	$ 12
Residential whole loans | Level 3 | Liquidation model | Minimum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	6.80%
Liquidation timeline (in years)	2 years
Fair Value Inputs, Value of Underlying Property	$ 124
Residential whole loans | Level 3 | Liquidation model | Maximum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	12.10%	6.50%
Liquidation timeline (in years)	4 years 9 months 18 days	4 years 9 months 18 days
Fair Value Inputs, Value of Underlying Property	$ 3,704	$ 4,500
Residential whole loans | Level 3 | Liquidation model | Maximum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	8.70%
Liquidation timeline (in years)	3 years 3 months 18 days
Fair Value Inputs, Value of Underlying Property	$ 1,750
Discount rate | Residential whole loans | Level 3 | Discounted cash flow | Minimum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.025	0.033
Discount rate | Residential whole loans | Level 3 | Discounted cash flow | Minimum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.008
Discount rate | Residential whole loans | Level 3 | Discounted cash flow | Maximum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.118	0.080
Discount rate | Residential whole loans | Level 3 | Discounted cash flow | Maximum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.073
Discount rate | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.035	0.039
Discount rate | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.031
Discount rate | Residential whole loans | Level 3 | Liquidation model | Minimum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.067	0.067
Discount rate | Residential whole loans | Level 3 | Liquidation model | Minimum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.075
Discount rate | Residential whole loans | Level 3 | Liquidation model | Maximum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.500	0.500
Discount rate | Residential whole loans | Level 3 | Liquidation model | Maximum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.075
Discount rate | Residential whole loans | Level 3 | Liquidation model | Weighted Average | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.080	0.081
Discount rate | Residential whole loans | Level 3 | Liquidation model | Weighted Average | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.075
Prepayment rate | Residential whole loans | Level 3 | Discounted cash flow | Minimum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Prepayment rate | Residential whole loans | Level 3 | Discounted cash flow | Minimum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000
Prepayment rate | Residential whole loans | Level 3 | Discounted cash flow | Maximum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.439	0.176
Prepayment rate | Residential whole loans | Level 3 | Discounted cash flow | Maximum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.461
Prepayment rate | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.140	0.054
Prepayment rate | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.202
Prepayment rate | Residential whole loans | Level 3 | Liquidation model | Weighted Average | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	7.00%	3.60%
Prepayment rate | Residential whole loans | Level 3 | Liquidation model | Weighted Average | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	8.10%
Default rate | Residential whole loans | Level 3 | Discounted cash flow | Minimum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Default rate | Residential whole loans | Level 3 | Discounted cash flow | Minimum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000
Default rate | Residential whole loans | Level 3 | Discounted cash flow | Maximum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.491	0.477
Default rate | Residential whole loans | Level 3 | Discounted cash flow | Maximum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.221
Default rate | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.041	0.041
Default rate | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.004
Default rate | Residential whole loans | Level 3 | Liquidation model | Weighted Average | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Liquidation timeline (in years)	1 year 9 months 18 days	1 year 9 months 18 days
Default rate | Residential whole loans | Level 3 | Liquidation model | Weighted Average | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Liquidation timeline (in years)	2 years 6 months
Loss severity | Residential whole loans | Level 3 | Discounted cash flow | Minimum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Loss severity | Residential whole loans | Level 3 | Discounted cash flow | Minimum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000
Loss severity | Residential whole loans | Level 3 | Discounted cash flow | Maximum | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	1.000	1.000
Loss severity | Residential whole loans | Level 3 | Discounted cash flow | Maximum | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.100
Loss severity | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.120	0.127
Loss severity | Residential whole loans | Level 3 | Discounted cash flow | Weighted Average | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.092
Loss severity | Residential whole loans | Level 3 | Liquidation model | Weighted Average | Nonperforming
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Value of Underlying Property	$ 765	$ 729
Loss severity | Residential whole loans | Level 3 | Liquidation model | Weighted Average | Performing
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Value of Underlying Property	$ 1,048