Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details)	6 Months Ended
	Jun. 30, 2023 USD ($) $ / loan	Dec. 31, 2022 USD ($)
Assets
ABS issued, net, At fair value | $	$ 7,702,826,000	$ 7,424,132,000
Variable Interest Entity, Primary Beneficiary | Collateralized Financing Entities | HEI
Liabilities
Fair value of securities owned | $	14,000,000
ABS Issued
Liabilities
ABS issued | $	$ 3,647,439,000
ABS Issued | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.04
ABS Issued | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.18
ABS Issued | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.07
ABS Issued | Default rate | Minimum
Liabilities
ABS issued, measurement input	0
ABS Issued | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.14
ABS Issued | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.01
ABS Issued | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.25
ABS Issued | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.50
ABS Issued | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.31
ABS Issued | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.04
ABS Issued | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.35
ABS Issued | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.10
CAFL
Liabilities
ABS issued | $	$ 2,475,176,000
CAFL | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0
CAFL | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.12
CAFL | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.07
CAFL | Default rate | Minimum
Liabilities
ABS issued, measurement input	0.05
CAFL | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.14
CAFL | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.07
CAFL | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.30
CAFL | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.40
CAFL | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.30
CAFL | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0
CAFL | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.03
CAFL | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.001
Freddie Mac SLST
Liabilities
ABS issued | $	$ 1,096,972,000
Freddie Mac SLST | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.05
Freddie Mac SLST | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.16
Freddie Mac SLST | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Default rate | Minimum
Liabilities
ABS issued, measurement input	0.08
Freddie Mac SLST | Default rate | Maximum
Liabilities
ABS issued, measurement input	0.09
Freddie Mac SLST | Default rate | Weighted Average
Liabilities
ABS issued, measurement input	0.09
Freddie Mac SLST | Loss severity | Minimum
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST | Loss severity | Maximum
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST | Loss severity | Weighted Average
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.06
Freddie Mac K-Series
Liabilities
ABS issued | $	$ 387,581,000
Freddie Mac K-Series | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.03
Freddie Mac K-Series | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.10
Freddie Mac K-Series | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.06
Consolidated Point Entities
Liabilities
ABS issued | $	$ 95,658,000
Consolidated Point Entities | Whole loan discount rate | Minimum
Liabilities
ABS issued, measurement input	0.10
Consolidated Point Entities | Whole loan discount rate | Maximum
Liabilities
ABS issued, measurement input	0.14
Consolidated Point Entities | Whole loan discount rate | Weighted Average
Liabilities
ABS issued, measurement input	0.10
Consolidated Point Entities | Prepayment rate (annual CPR) | Minimum
Liabilities
ABS issued, measurement input	0.20
Consolidated Point Entities | Prepayment rate (annual CPR) | Maximum
Liabilities
ABS issued, measurement input	0.20
Consolidated Point Entities | Prepayment rate (annual CPR) | Weighted Average
Liabilities
ABS issued, measurement input	0.20
Consolidated Point Entities | Home price appreciation (depreciation) | Minimum
Liabilities
ABS issued, measurement input	(0.03)
Consolidated Point Entities | Home price appreciation (depreciation) | Maximum
Liabilities
ABS issued, measurement input	0.04
Consolidated Point Entities | Home price appreciation (depreciation) | Weighted Average
Liabilities
ABS issued, measurement input	0.03
Jumbo loans
Assets
ABS issued, net, At fair value | $	$ 182,383,000
Jumbo loans | Whole loan spread to TBA price | Minimum
Assets
Loans held-for-sale, measurement input	3.91
Jumbo loans | Whole loan spread to TBA price | Maximum
Assets
Loans held-for-sale, measurement input	3.91
Jumbo loans | Whole loan spread to TBA price | Weighted Average
Assets
Loans held-for-sale, measurement input	3.91
Jumbo loans committed to sell
Assets
ABS issued, net, At fair value | $	$ 14,318,000
Jumbo loans committed to sell | Whole loan committed sales price | Minimum
Assets
Residential loans, measurement input	100
Jumbo loans committed to sell | Whole loan committed sales price | Maximum
Assets
Residential loans, measurement input	102
Jumbo loans committed to sell | Whole loan committed sales price | Weighted Average
Assets
Residential loans, measurement input	101
Legacy Sequoia
Assets
Loans receivable, fair value | $	$ 163,222,000
Sequoia
Assets
Loans receivable, fair value | $	3,703,754,000
Liabilities
Fair value of securities owned | $	218,000,000
Freddie Mac SLST
Assets
Loans receivable, fair value | $	1,392,186,000
Liabilities
Fair value of securities owned | $	298,000,000	323,000,000
BPL term loans
Assets
Loans receivable, fair value | $	$ 269,886,000
BPL term loans | Senior credit spread | Minimum
Assets
Loans receivable, measurement input	0.0185
BPL term loans | Senior credit spread | Maximum
Assets
Loans receivable, measurement input	0.0185
BPL term loans | Senior credit spread | Weighted Average
Assets
Loans receivable, measurement input	0.0185
BPL term loans | Subordinate credit spread | Minimum
Assets
Loans receivable, measurement input	0.0275
BPL term loans | Subordinate credit spread | Maximum
Assets
Loans receivable, measurement input	0.0915
BPL term loans | Subordinate credit spread | Weighted Average
Assets
Loans receivable, measurement input	0.0491
BPL term loans | Senior credit support | Minimum
Assets
Loans receivable, measurement input	0.36
BPL term loans | Senior credit support | Maximum
Assets
Loans receivable, measurement input	0.36
BPL term loans | Senior credit support | Weighted Average
Assets
Loans receivable, measurement input	0.36
BPL term loans | IO discount rate | Minimum
Assets
Loans receivable, measurement input	0.08
BPL term loans | IO discount rate | Maximum
Assets
Loans receivable, measurement input	0.08
BPL term loans | IO discount rate | Weighted Average
Assets
Loans receivable, measurement input	0.08
BPL term loans | Dollar price of NPLs | Minimum
Assets
Loans receivable, measurement input	58
BPL term loans | Dollar price of NPLs | Maximum
Assets
Loans receivable, measurement input	100
BPL term loans | Dollar price of NPLs | Weighted Average
Assets
Loans receivable, measurement input	59
BPL term loans | Prepayment rate (annual CPR) | Minimum
Assets
Loans receivable, measurement input	0
BPL term loans | Prepayment rate (annual CPR) | Maximum
Assets
Loans receivable, measurement input	0.03
BPL term loans | Prepayment rate (annual CPR) | Weighted Average
Assets
Loans receivable, measurement input	0.03
CAFL
Assets
Loans receivable, fair value | $	$ 2,783,731,000
Liabilities
Fair value of securities owned | $	306,000,000	$ 304,000,000
BPL bridge loans
Assets
Loans receivable, fair value | $	$ 2,173,106
BPL bridge loans | Whole loan discount rate | Minimum
Assets
Loans held-for-investment, measurement input	0.04
BPL bridge loans | Whole loan discount rate | Maximum
Assets
Loans held-for-investment, measurement input	0.15
BPL bridge loans | Whole loan discount rate | Weighted Average
Assets
Loans held-for-investment, measurement input	0.09
BPL bridge loans | Whole loan spread | Minimum
Assets
Loans held-for-investment, measurement input	0.0545
BPL bridge loans | Whole loan spread | Maximum
Assets
Loans held-for-investment, measurement input	0.0545
BPL bridge loans | Whole loan spread | Weighted Average
Assets
Loans held-for-investment, measurement input	0.0545
Freddie Mac K-Series
Assets
Loans receivable, fair value | $	$ 420,096,000
Liabilities
Fair value of securities owned | $	33,000,000
Trading and AFS securities
Assets
Trading and AFS securities | $	$ 166,819,000
Trading and AFS securities | Whole loan discount rate | Minimum
Assets
Trading and AFS securities	0.06
Trading and AFS securities | Whole loan discount rate | Maximum
Assets
Trading and AFS securities	0.18
Trading and AFS securities | Whole loan discount rate | Weighted Average
Assets
Trading and AFS securities	0.09
Trading and AFS securities | Default rate | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Default rate | Maximum
Assets
Trading and AFS securities	0.14
Trading and AFS securities | Default rate | Weighted Average
Assets
Trading and AFS securities	0.004
Trading and AFS securities | Loss severity | Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities | Loss severity | Maximum
Assets
Trading and AFS securities	0.50
Trading and AFS securities | Loss severity | Weighted Average
Assets
Trading and AFS securities	0.23
Trading and AFS securities | CRT dollar price | Minimum
Assets
Trading and AFS securities	97
Trading and AFS securities | CRT dollar price | Maximum
Assets
Trading and AFS securities	99
Trading and AFS securities | CRT dollar price | Weighted Average
Assets
Trading and AFS securities	98
Trading and AFS securities | Prepayment rate (annual CPR) | Minimum
Assets
Trading and AFS securities	0.04
Trading and AFS securities | Prepayment rate (annual CPR) | Maximum
Assets
Trading and AFS securities	0.65
Trading and AFS securities | Prepayment rate (annual CPR) | Weighted Average
Assets
Trading and AFS securities	0.09
HEIs
Assets
HEIs | $	$ 298,043,000
HEIs | Whole loan discount rate | Minimum
Assets
HEIs, Measurement input	0.10
HEIs | Whole loan discount rate | Maximum
Assets
HEIs, Measurement input	0.10
HEIs | Whole loan discount rate | Weighted Average
Assets
HEIs, Measurement input	0.10
HEIs | Prepayment rate (annual CPR) | Minimum
Assets
HEIs, Measurement input	0.01
HEIs | Prepayment rate (annual CPR) | Maximum
Assets
HEIs, Measurement input	0.23
HEIs | Prepayment rate (annual CPR) | Weighted Average
Assets
HEIs, Measurement input	0.16
HEIs | Home price appreciation (depreciation) | Minimum
Assets
HEIs, Measurement input	(0.03)
HEIs | Home price appreciation (depreciation) | Maximum
Assets
HEIs, Measurement input	0.04
HEIs | Home price appreciation (depreciation) | Weighted Average
Assets
HEIs, Measurement input	0.03
HEIs held by Point HEI entity
Assets
HEIs | $	$ 129,264,000
Servicer Advance Investments
Assets
Servicing asset | $	$ 234,304,000
Servicer Advance Investments | Minimum
Assets
Expected remaining life	6 years
Servicer Advance Investments | Maximum
Assets
Expected remaining life	6 years
Servicer Advance Investments | Weighted Average
Assets
Expected remaining life	6 years
Servicer Advance Investments | Whole loan discount rate | Minimum
Assets
Servicing asset, measurement input	0.02
Servicer Advance Investments | Whole loan discount rate | Maximum
Assets
Servicing asset, measurement input	0.05
Servicer Advance Investments | Whole loan discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.04
Servicer Advance Investments | Mortgage servicing income | Minimum
Assets
Servicing asset, measurement input	0
Servicer Advance Investments | Mortgage servicing income | Maximum
Assets
Servicing asset, measurement input	0.0018
Servicer Advance Investments | Mortgage servicing income | Weighted Average
Assets
Servicing asset, measurement input	0.0003
Servicer Advance Investments | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.11
Servicer Advance Investments | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.30
Servicer Advance Investments | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.14
MSRs
Assets
Servicing asset | $	$ 26,242,000
MSRs | Whole loan discount rate | Minimum
Assets
Servicing asset, measurement input	0.11
MSRs | Whole loan discount rate | Maximum
Assets
Servicing asset, measurement input	0.46
MSRs | Whole loan discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.11
MSRs | Per loan annual cost to service | Minimum
Assets
Servicing asset, measurement input	93
MSRs | Per loan annual cost to service | Maximum
Assets
Servicing asset, measurement input	93
MSRs | Per loan annual cost to service | Weighted Average
Assets
Servicing asset, measurement input	93
MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.04
MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	0.25
MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.07
Excess MSRs
Assets
Servicing asset | $	$ 39,877,000
Excess MSRs | Whole loan discount rate | Minimum
Assets
Servicing asset, measurement input	0.13
Excess MSRs | Whole loan discount rate | Maximum
Assets
Servicing asset, measurement input	0.19
Excess MSRs | Whole loan discount rate | Weighted Average
Assets
Servicing asset, measurement input	0.18
Excess MSRs | Excess mortgage servicing amount | Minimum
Assets
Servicing asset, measurement input	0.0008
Excess MSRs | Excess mortgage servicing amount | Maximum
Assets
Servicing asset, measurement input	0.0020
Excess MSRs | Excess mortgage servicing amount | Weighted Average
Assets
Servicing asset, measurement input	0.0011
Excess MSRs | Prepayment rate (annual CPR) | Minimum
Assets
Servicing asset, measurement input	0.10
Excess MSRs | Prepayment rate (annual CPR) | Maximum
Assets
Servicing asset, measurement input	1
Excess MSRs | Prepayment rate (annual CPR) | Weighted Average
Assets
Servicing asset, measurement input	0.17
Residential loan purchase commitments
Assets
Loan purchase commitments, net | $	$ 2,046,000
Residential loan purchase commitments | Whole loan spread to TBA price | Minimum
Assets
Loans receivable, measurement input	3.91
Residential loan purchase commitments | Whole loan spread to TBA price | Maximum
Assets
Loans receivable, measurement input	3.91
Residential loan purchase commitments | Whole loan spread to TBA price | Weighted Average
Assets
Loans receivable, measurement input	3.91
Residential loan purchase commitments | Pull-through rate | Minimum
Assets
Loans receivable, measurement input	0.17
Residential loan purchase commitments | Pull-through rate | Maximum
Assets
Loans receivable, measurement input	1
Residential loan purchase commitments | Pull-through rate | Weighted Average
Assets
Loans receivable, measurement input	0.68
Residential loan purchase commitments | Committed sales price | Minimum
Assets
Loans receivable, measurement input	100
Residential loan purchase commitments | Committed sales price | Maximum
Assets
Loans receivable, measurement input	103
Residential loan purchase commitments | Committed sales price | Weighted Average
Assets
Loans receivable, measurement input	101