Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Amortized Cost in Loans Held for Investment based on Accrual Status and by Loan Origination Year (Details) - Accrual Status - USD ($) $ in Thousands	Jun. 30, 2022	Dec. 31, 2021
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	$ 894,190	$ 1,143,849
Term Loans Amortized Cost Basis by Origination Year, 2021	1,091,939	214,490
Term Loans Amortized Cost Basis by Origination Year, 2020	181,730	500,689
Term Loans Amortized Cost Basis by Origination Year, 2019	411,977	326,082
Term Loans Amortized Cost Basis by Origination Year, 2018	262,433	171,370
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	281,435	175,345
Total	3,123,704	2,531,825
Commercial Purchase Portfolio Segment
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	215,509	277,906
Term Loans Amortized Cost Basis by Origination Year, 2021	269,213	47,617
Term Loans Amortized Cost Basis by Origination Year, 2020	41,777	87,082
Term Loans Amortized Cost Basis by Origination Year, 2019	70,767	50,817
Term Loans Amortized Cost Basis by Origination Year, 2018	41,483	27,703
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	38,742	18,296
Total	677,491	509,421
Commercial Purchase Portfolio Segment | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	215,509	277,618
Term Loans Amortized Cost Basis by Origination Year, 2021	268,001	45,836
Term Loans Amortized Cost Basis by Origination Year, 2020	41,591	81,541
Term Loans Amortized Cost Basis by Origination Year, 2019	64,227	46,637
Term Loans Amortized Cost Basis by Origination Year, 2018	39,948	24,164
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	33,304	16,365
Total	662,580	492,161
Commercial Purchase Portfolio Segment | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022		288
Term Loans Amortized Cost Basis by Origination Year, 2021	1,212	1,781
Term Loans Amortized Cost Basis by Origination Year, 2020	186	5,541
Term Loans Amortized Cost Basis by Origination Year, 2019	6,540	4,180
Term Loans Amortized Cost Basis by Origination Year, 2018	1,535	3,539
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	5,438	1,931
Total	14,911	17,260
Commercial Refinance Portfolio Segment
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	224,604	242,170
Term Loans Amortized Cost Basis by Origination Year, 2021	233,369	68,915
Term Loans Amortized Cost Basis by Origination Year, 2020	61,943	170,029
Term Loans Amortized Cost Basis by Origination Year, 2019	145,747	145,604
Term Loans Amortized Cost Basis by Origination Year, 2018	121,399	78,866
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	125,712	77,676
Total	912,774	783,260
Commercial Refinance Portfolio Segment | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	224,064	239,688
Term Loans Amortized Cost Basis by Origination Year, 2021	225,103	64,966
Term Loans Amortized Cost Basis by Origination Year, 2020	58,086	144,017
Term Loans Amortized Cost Basis by Origination Year, 2019	124,285	118,735
Term Loans Amortized Cost Basis by Origination Year, 2018	100,753	62,374
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	100,041	67,545
Total	832,332	697,325
Commercial Refinance Portfolio Segment | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	540	2,482
Term Loans Amortized Cost Basis by Origination Year, 2021	8,266	3,949
Term Loans Amortized Cost Basis by Origination Year, 2020	3,857	26,012
Term Loans Amortized Cost Basis by Origination Year, 2019	21,462	26,869
Term Loans Amortized Cost Basis by Origination Year, 2018	20,646	16,492
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	25,671	10,131
Total	80,442	85,935
Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	163,477	264,552
Term Loans Amortized Cost Basis by Origination Year, 2021	250,607	15,627
Term Loans Amortized Cost Basis by Origination Year, 2020	13,710	52,826
Term Loans Amortized Cost Basis by Origination Year, 2019	42,775	33,280
Term Loans Amortized Cost Basis by Origination Year, 2018	25,979	16,350
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	33,248	26,135
Total	529,796	408,770
Residential 1-4 Unit - Purchase | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	162,492	263,180
Term Loans Amortized Cost Basis by Origination Year, 2021	242,544	12,878
Term Loans Amortized Cost Basis by Origination Year, 2020	11,541	48,930
Term Loans Amortized Cost Basis by Origination Year, 2019	39,733	29,544
Term Loans Amortized Cost Basis by Origination Year, 2018	22,505	12,863
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	28,096	23,990
Total	506,911	391,385
Residential 1-4 Unit - Purchase | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	985	1,372
Term Loans Amortized Cost Basis by Origination Year, 2021	8,063	2,749
Term Loans Amortized Cost Basis by Origination Year, 2020	2,169	3,896
Term Loans Amortized Cost Basis by Origination Year, 2019	3,042	3,736
Term Loans Amortized Cost Basis by Origination Year, 2018	3,474	3,487
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	5,152	2,145
Total	22,885	17,385
Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	248,479	354,845
Term Loans Amortized Cost Basis by Origination Year, 2021	329,431	37,374
Term Loans Amortized Cost Basis by Origination Year, 2020	33,261	145,961
Term Loans Amortized Cost Basis by Origination Year, 2019	124,390	90,451
Term Loans Amortized Cost Basis by Origination Year, 2018	68,426	48,451
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	83,733	53,238
Total	887,720	730,320
Residential 1-4 Unit - Refinance | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	247,970	343,199
Term Loans Amortized Cost Basis by Origination Year, 2021	314,865	31,334
Term Loans Amortized Cost Basis by Origination Year, 2020	26,150	114,145
Term Loans Amortized Cost Basis by Origination Year, 2019	95,470	59,825
Term Loans Amortized Cost Basis by Origination Year, 2018	44,360	31,774
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	62,256	42,491
Total	791,071	622,768
Residential 1-4 Unit - Refinance | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	509	11,646
Term Loans Amortized Cost Basis by Origination Year, 2021	14,566	6,040
Term Loans Amortized Cost Basis by Origination Year, 2020	7,111	31,816
Term Loans Amortized Cost Basis by Origination Year, 2019	28,920	30,626
Term Loans Amortized Cost Basis by Origination Year, 2018	24,066	16,677
Term Loans Amortized Cost Basis by Origination Year, Pre-2018	21,477	10,747
Total	96,649	107,552
Short Term 1-4 Unit - Purchase
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	29,097	1,890
Term Loans Amortized Cost Basis by Origination Year, 2021	7,493	17,147
Term Loans Amortized Cost Basis by Origination Year, 2020	16,032	9,847
Term Loans Amortized Cost Basis by Origination Year, 2019	5,246	105
Term Loans Amortized Cost Basis by Origination Year, 2018	105
Total	57,973	28,989
Short Term 1-4 Unit - Purchase | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	29,097	1,890
Term Loans Amortized Cost Basis by Origination Year, 2021	7,493	15,582
Term Loans Amortized Cost Basis by Origination Year, 2020	15,037	8,531
Term Loans Amortized Cost Basis by Origination Year, 2019	4,044
Total	55,671	26,003
Short Term 1-4 Unit - Purchase | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2021		1,565
Term Loans Amortized Cost Basis by Origination Year, 2020	995	1,316
Term Loans Amortized Cost Basis by Origination Year, 2019	1,202	105
Term Loans Amortized Cost Basis by Origination Year, 2018	105
Total	2,302	2,986
Short Term 1-4 Unit - Refinance
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	13,024	2,486
Term Loans Amortized Cost Basis by Origination Year, 2021	1,826	27,810
Term Loans Amortized Cost Basis by Origination Year, 2020	15,007	34,944
Term Loans Amortized Cost Basis by Origination Year, 2019	23,052	5,825
Term Loans Amortized Cost Basis by Origination Year, 2018	5,041
Total	57,950	71,065
Short Term 1-4 Unit - Refinance | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022	13,024	1,448
Term Loans Amortized Cost Basis by Origination Year, 2021	1,582	11,991
Term Loans Amortized Cost Basis by Origination Year, 2020	2,543	12,326
Term Loans Amortized Cost Basis by Origination Year, 2019	2,788
Total	19,937	25,765
Short Term 1-4 Unit - Refinance | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2022		1,038
Term Loans Amortized Cost Basis by Origination Year, 2021	244	15,819
Term Loans Amortized Cost Basis by Origination Year, 2020	12,464	22,618
Term Loans Amortized Cost Basis by Origination Year, 2019	20,264	5,825
Term Loans Amortized Cost Basis by Origination Year, 2018	5,041
Total	$ 38,013	$ 45,300