Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Nonaccrual With No Allowance for Loan Loss and Total Nonaccrual of Loans Held for Investment (Details) - USD ($) $ in Thousands	Mar. 31, 2024	Dec. 31, 2023	Mar. 31, 2023
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 327,564	$ 325,354
Nonaccrual with No Allowance for Loan Loss	313,215	315,906
Nonaccrual with Allowance for Loan Loss	14,349	9,449
Allowance for Loans Individually Evaluated	1,431	974	$ 996
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	28,127	28,221
Nonaccrual with No Allowance for Loan Loss	26,843	27,037
Nonaccrual with Allowance for Loan Loss	1,284	1,184
Allowance for Loans Individually Evaluated	130	156	138
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	97,571	86,890
Nonaccrual with No Allowance for Loan Loss	93,511	84,575
Nonaccrual with Allowance for Loan Loss	4,060	2,315
Allowance for Loans Individually Evaluated	642	444	416
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	38,090	36,253
Nonaccrual with No Allowance for Loan Loss	36,962	36,253
Nonaccrual with Allowance for Loan Loss	1,128
Allowance for Loans Individually Evaluated	185
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	130,052	137,925
Nonaccrual with No Allowance for Loan Loss	124,779	134,579
Nonaccrual with Allowance for Loan Loss	5,273	3,346
Allowance for Loans Individually Evaluated	253	245	204
Short Term 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	5,584	6,402
Nonaccrual with No Allowance for Loan Loss	5,584	6,403
Short Term 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	28,140	29,663
Nonaccrual with No Allowance for Loan Loss	25,536	27,059
Nonaccrual with Allowance for Loan Loss	2,604	2,604
Allowance for Loans Individually Evaluated	$ 221	$ 129	$ 238