Fair Value of Financial Instruments - Recorded Amount of Assets and Liabilities Measured at Fair Value on a Recurring Basis (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	$ 750,098	$ 540,045
Loans held for sale	36,111	16,198
Loans Accounted for Under the Fair Value Option	815,374	824,520
Servicing assets	33,918	35,365	$ 47,641
US government agencies
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	15,919	22,779
Mortgage-backed securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	730,454	503,297
US Treasury Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale		5,015
Recurring
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Loans held for sale	36,111	16,198
Loans Accounted for Under the Fair Value Option	815,374	824,520
Servicing assets	33,918	35,365
Equity warrant assets	908	570
Total assets at fair value	1,638,760	1,418,904
Recurring | Municipal bonds
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	3,725	8,954
Recurring | US government agencies
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	15,919	22,779
Recurring | Mortgage-backed securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	730,454	503,297
Recurring | Mutual Fund
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mutual fund	2,351	2,206
Recurring | US Treasury Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale		5,015
Recurring | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Loans held for sale	0	0
Loans Accounted for Under the Fair Value Option	0	0
Servicing assets	0	0
Equity warrant assets	0	0
Total assets at fair value	0	0
Recurring | Level 1 | Municipal bonds
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 1 | US government agencies
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 1 | Mortgage-backed securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 1 | Mutual Fund
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mutual fund	0	0
Recurring | Level 1 | US Treasury Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale		0
Recurring | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Loans held for sale	0	0
Loans Accounted for Under the Fair Value Option	0	0
Servicing assets	0	0
Equity warrant assets	0	0
Total assets at fair value	752,353	542,159
Recurring | Level 2 | Municipal bonds
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	3,629	8,862
Recurring | Level 2 | US government agencies
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	15,919	22,779
Recurring | Level 2 | Mortgage-backed securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	730,454	503,297
Recurring | Level 2 | Mutual Fund
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mutual fund	2,351	2,206
Recurring | Level 2 | US Treasury Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale		5,015
Recurring | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Loans held for sale	36,111	16,198
Loans Accounted for Under the Fair Value Option	815,374	824,520
Servicing assets	33,918	35,365
Equity warrant assets	908	570
Total assets at fair value	886,407	876,745
Recurring | Level 3 | Municipal bonds
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	96	92
Total assets at fair value	96	92
Recurring | Level 3 | US government agencies
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 3 | Mortgage-backed securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale	0	0
Recurring | Level 3 | Mutual Fund
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mutual fund	$ 0	0
Recurring | Level 3 | US Treasury Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Investment securities available-for-sale		$ 0