Fair Value of Financial Instruments - Fair Value Methodology (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2021 USD ($)	Dec. 31, 2020 USD ($)
Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	$ 1,071,774	$ 1,216,637
Purchases excluded from level 2 fair value	496	265
Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	3,151,876
Discounted cash flow | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	720,766	789,576
Discounted cash flow | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	3,143,928
Liquidation model | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	351,008	427,061
Simple average amount	421	$ 380
Liquidation model | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Purchased Loans	$ 7,948
Residential whole loans, at fair value | Level 3 | Liquidation model | Minimum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	4.50%	0.00%
Liquidation timeline (in years)	1 month 6 days	9 months 18 days
Fair Value Inputs, Value of Underlying Property	$ 10	$ 12
Residential whole loans, at fair value | Level 3 | Liquidation model | Minimum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	0.00%
Liquidation timeline (in years)	9 months 18 days
Fair Value Inputs, Value of Underlying Property	$ 60
Residential whole loans, at fair value | Level 3 | Liquidation model | Maximum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	21.90%	6.50%
Liquidation timeline (in years)	4 years 6 months	4 years 9 months 18 days
Fair Value Inputs, Value of Underlying Property	$ 3,995	$ 4,500
Residential whole loans, at fair value | Level 3 | Liquidation model | Maximum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	14.80%
Liquidation timeline (in years)	4 years 2 months 12 days
Fair Value Inputs, Value of Underlying Property	$ 1,750
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.015	0.033
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.014
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.098	0.080
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.259
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.036	0.039
Discount rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.039
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Minimum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.067	0.067
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Minimum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.070
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Maximum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.500	0.500
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Maximum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.070
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.080	0.081
Discount rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.070
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.440	0.099
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.472
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.144	0.048
Prepayment rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.190
Prepayment rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	9.70%	3.60%
Prepayment rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Annual Change in Home Prices	6.50%
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.508	0.189
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.178
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.039	0.038
Default rate | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.002
Default rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Liquidation timeline (in years)	1 year 8 months 12 days	1 year 9 months 18 days
Default rate | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Liquidation timeline (in years)	2 years
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000	0.000
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Minimum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.000
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	1.000	1.000
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Maximum | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.100
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.117	0.127
Loss severity | Residential whole loans, at fair value | Level 3 | Discounted cash flow | Weighted Average | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Measurement input	0.084
Loss severity | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average | Nonperforming Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Value of Underlying Property	$ 770	$ 729
Loss severity | Residential whole loans, at fair value | Level 3 | Liquidation model | Weighted Average | Performing Financial Instruments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Fair Value Inputs, Value of Underlying Property	$ 691