Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Nonaccrual With No Allowance for Loan Loss and Total Nonaccrual of Loans Held for Investment (Details) - USD ($) $ in Thousands	Jun. 30, 2024	Dec. 31, 2023	Jun. 30, 2023
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 327,290	$ 325,354
Nonaccrual with No Allowance for Credit Loss	309,403	315,905
Nonaccrual with Allowance for Credit Loss	17,887	9,449
Allowance for Loans Individually Evaluated	1,614	974	$ 1,043
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	30,198	28,221
Nonaccrual with No Allowance for Credit Loss	29,162	27,037
Nonaccrual with Allowance for Credit Loss	1,036	1,184
Allowance for Loans Individually Evaluated	111	156	137
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	99,844	86,890
Nonaccrual with No Allowance for Credit Loss	94,067	84,575
Nonaccrual with Allowance for Credit Loss	5,777	2,315
Allowance for Loans Individually Evaluated	552	444	552
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	33,640	36,253
Nonaccrual with No Allowance for Credit Loss	32,411	36,253
Nonaccrual with Allowance for Credit Loss	1,229
Allowance for Loans Individually Evaluated	189
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	132,448	137,925
Nonaccrual with No Allowance for Credit Loss	126,721	134,579
Nonaccrual with Allowance for Credit Loss	5,727	3,346
Allowance for Loans Individually Evaluated	273	245	246
Short Term 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	6,904	6,402
Nonaccrual with No Allowance for Credit Loss	6,904	6,402
Short Term 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	24,256	29,663
Nonaccrual with No Allowance for Credit Loss	20,138	27,059
Nonaccrual with Allowance for Credit Loss	4,118	2,604
Allowance for Loans Individually Evaluated	$ 489	$ 129	$ 108