Segment Information - Balance Sheet (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Assets:
Cash and cash equivalents	$ 339,528	$ 299,687	$ 160,063	$ 104,374
Restricted cash	197,470	210,875	$ 180,606	$ 139,398
Loans and investments, net	5,285,868	4,189,960
Loans held-for-sale, net	986,919	861,360
Capitalized mortgage servicing rights, net	379,974	286,420
Securities held-to-maturity, net	95,524	88,699
Investments in equity affiliates	74,274	41,800
Goodwill and other intangible assets	105,451	110,700
Other assets	195,978	149,659
Total assets	7,660,986	6,239,160
Liabilities:
Debt obligations	5,824,664	4,621,938
Allowance for loss-sharing obligations	64,303	34,648
Other liabilities	289,334	226,547
Total liabilities	6,178,301	4,883,133
Structured Business | Operating segments
Assets:
Cash and cash equivalents	172,568	264,468
Restricted cash	188,226	208,926
Loans and investments, net	5,285,868	4,189,960
Securities held-to-maturity, net		20,000
Investments in equity affiliates	74,274	41,800
Goodwill and other intangible assets	12,500	12,500
Other assets	142,844	118,175
Total assets	5,876,280	4,855,829
Liabilities:
Debt obligations	4,872,626	3,878,343
Other liabilities	203,554	171,004
Total liabilities	5,076,180	4,049,347
Agency Business | Operating segments
Assets:
Cash and cash equivalents	166,960	35,219
Restricted cash	9,244	1,949
Loans held-for-sale, net	986,919	861,360
Capitalized mortgage servicing rights, net	379,974	286,420
Securities held-to-maturity, net	95,524	68,699
Goodwill and other intangible assets	92,951	98,200
Other assets	53,134	31,484
Total assets	1,784,706	1,383,331
Liabilities:
Debt obligations	952,038	743,595
Allowance for loss-sharing obligations	64,303	34,648
Other liabilities	85,780	55,543
Total liabilities	$ 1,102,121	$ 833,786