Summary of Significant Accounting Policies (Details) $ in Thousands	3 Months Ended				12 Months Ended
	Dec. 31, 2020 USD ($)	Sep. 30, 2020 USD ($)	Jun. 30, 2020 USD ($)	Mar. 31, 2020 USD ($)	Dec. 31, 2020 USD ($) segment	Dec. 31, 2019 USD ($)	Dec. 31, 2018 USD ($)	Dec. 31, 2017 USD ($)	Jan. 23, 2017
Accounts, Notes, Loans and Financing Receivable [Line Items]
Goodwill	$ 0				$ 0	$ 25,222
Impairment of goodwill	0	$ 0	$ 0	$ 25,222	$ 25,222	0	$ 0
Residential mortgage loans, delinquency period					90 days
Restricted cash and cash equivalents	11,307				$ 11,307	2,849	5,421
Receivables, loan services, recoverable advances and interest receivable						56,300
Operating lease right-of-use asset	10,100				10,100	9,300
Operating lease, liability, current	10,600				$ 10,600	9,800
Reportable segment | segment					1
Equity	$ 2,307,573				$ 2,307,573	2,205,029	1,180,293	$ 976,001
Cumulative Effect, Period of Adoption, Adjustment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Equity						12,284
VIE, Primary Beneficiary
Accounts, Notes, Loans and Financing Receivable [Line Items]
Impairment of real estate					$ 1,800	1,900	2,800
Interest rate	2.54%				2.54%
Performance Shares
Accounts, Notes, Loans and Financing Receivable [Line Items]
Award vesting period					3 years
6.25% senior convertible notes due 2022 | Convertible Notes
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest rate	6.25%				6.25%				6.25%
Net Income (Loss) Attributable to Noncontrolling Interest | VIE, Primary Beneficiary
Accounts, Notes, Loans and Financing Receivable [Line Items]
Impairment of real estate					$ 900	1,000	$ 1,400
Net Loss Attributable to Parent | VIE, Primary Beneficiary
Accounts, Notes, Loans and Financing Receivable [Line Items]
Impairment of real estate						900
Other Assets
Accounts, Notes, Loans and Financing Receivable [Line Items]
Inventory, real estate, construction in process	$ 0				0	14,500
Receivables, loan services, recoverable advances and interest receivable	$ 63,600				$ 63,600
Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Finite-lived intangible asset, useful life					6 months
Repurchase agreements, expiration period					23 months
Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Finite-lived intangible asset, useful life					10 years
Repurchase agreements, expiration period					8 months
Building
Accounts, Notes, Loans and Financing Receivable [Line Items]
Property, plant and equipment useful life					30 years
Building Improvements | Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Property, plant and equipment useful life					10 years
Building Improvements | Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Property, plant and equipment useful life					30 years
Furniture and Fixtures
Accounts, Notes, Loans and Financing Receivable [Line Items]
Property, plant and equipment useful life					5 years
RiverBanc, RBMI, and RBDHC
Accounts, Notes, Loans and Financing Receivable [Line Items]
Goodwill						$ 25,200