CONSOLIDATED STATEMENTS OF CASH FLOWS $ in Thousands	12 Months Ended
	Dec. 31, 2020 USD ($)	Dec. 31, 2019 USD ($)	Dec. 31, 2018 USD ($)
Operating activities:
Net income	$ 196,157	$ 155,238	$ 148,051
Adjustments to reconcile net income to net cash provided by (used in) operating activities:
Depreciation and amortization	7,640	7,510	7,453
Stock-based compensation	9,046	9,515	6,095
Amortization and accretion of interest and fees, net	2,964	5,045	9,291
Amortization of capitalized mortgage servicing rights	49,222	48,681	48,124
Impairment loss on real estate owned		1,000	2,000
Originations of loans held-for-sale	(6,709,375)	(4,564,032)	(5,074,662)
Proceeds from sales of loans held-for-sale, net of gain on sale	6,587,728	4,189,787	4,893,886
Mortgage servicing rights	(165,517)	(90,761)	(98,839)
Write-off of capitalized mortgage servicing rights from payoffs	16,757	22,425	25,058
Provision for credit losses (net of recoveries)	61,110		8,353
Provision for loss sharing (net of recoveries)	14,822	1,147	3,843
Net recoveries (charge-offs) for loss sharing obligations	427	(797)	(56)
Deferred tax provision (benefit)	4,726	150	(12,033)
Income from equity affiliates	(76,161)	(10,635)	(1,196)
Distributions from equity affiliates	44,579	3,724	2,640
Loss on extinguishment of debt	3,546	7,439	5,041
Payoffs and paydowns of loans held-for-sale	179	84	61
Loss on sale of real estate	375
Changes in operating assets and liabilities	6,940	(12,057)	(10,845)
Net cash provided by (used in) operating activities	55,165	(226,537)	(37,735)
Investing Activities:
Loans and investments funded, originated and purchased, net	(2,376,233)	(2,712,354)	(1,545,499)
Payoffs and paydowns of loans and investments	1,243,694	1,753,691	960,769
Deferred fees	18,766	22,340	13,114
Investments in real estate, net	(131)	(475)	(367)
Proceeds from sale of real estate, net	8,870
Proceeds from sale of available-for-sale securities	9,995
Contributions to equity affiliates	(892)	(13,522)	(2,493)
Distributions from equity affiliates		213	3,122
Purchase of securities held-to-maturity, net	(37,926)	(20,000)	(47,499)
Payoffs and paydowns of securities held-to-maturity	10,158	12,488	2,269
Due to borrowers and reserves	(32,925)	(37,121)	(65,213)
Distributions of insurance settlements, net			(78)
Net cash used in investing activities	(1,156,624)	(994,740)	(681,875)
Financing activities:
Proceeds from repurchase agreements and credit facilities	12,986,256	8,986,286	9,166,255
Paydowns and payoffs of repurchase agreements and credit facilities	(12,428,681)	(8,443,275)	(8,559,057)
Proceeds from issuance of collateralized loan obligations	668,000	1,067,193	441,000
Payoffs and paydowns of collateralized loan obligations	(283,125)	(529,250)	(267,750)
Payoff of debt fund	(70,000)
Proceeds from issuance of common stock	183,585	260,383	156,417
Proceeds from issuance of convertible senior unsecured notes		264,000	264,500
Extinguishment of convertible senior unsecured notes	(22,336)	(231,940)	(228,287)
Settlements of convertible senior unsecured notes	(22,336)	(231,940)	(228,287)
Proceeds from issuance of senior unsecured notes	345,750	200,000	125,000
Payoffs of senior unsecured notes			(97,860)
Redemption of OP Units	(32,233)	(1,674)	(6,845)
Payments of withholding taxes on net settlement of vested stock	(6,594)	(3,634)
Repurchase of common stock	(21,531)	(11,574)	(10,066)
Distributions paid on common stock	(141,801)	(107,846)	(68,083)
Distributions paid on noncontrolling interest	(23,744)	(23,387)	(20,650)
Distributions paid on preferred stock	(7,564)	(7,569)	(7,568)
Payment of deferred financing costs	(18,087)	(26,543)	(20,499)
Payoff of related party financing			(50,000)
Net cash provided by financing activities	1,127,895	1,391,170	816,507
Net increase in cash, cash equivalents and restricted cash	26,436	169,893	96,897
Cash, cash equivalents and restricted cash at beginning of period	510,562	340,669	243,772
Cash, cash equivalents and restricted cash at end of period	536,998	510,562	340,669
Reconciliation of cash, cash equivalents and restricted cash:
Cash and cash equivalents at beginning of period	299,687	160,063	104,374
Restricted cash at beginning of period	210,875	180,606	139,398
Cash, cash equivalents and restricted cash at beginning of period	510,562	340,669	243,772
Cash and cash equivalents at end of period	339,528	299,687	160,063
Restricted cash at end of period	197,470	210,875	180,606
Cash, cash equivalents and restricted cash at end of period	536,998	510,562	340,669
Supplemental cash flow information:
Cash used to pay interest	144,968	167,581	118,923
Cash used to pay taxes	35,282	19,611	20,026
Supplemental schedule of non-cash investing and financing activities:
Cumulative-effect adjustment (Note 2)	(1,482,685)	(1,356,027)	(1,065,566)
Issuance of common stock from convertible debt	90	69,232	80,118
Extinguishment of convertible senior unsecured notes		(69,510)	(70,271)
Distributions accrued for special dividend declared			15,696
Settlements of convertible senior unsecured notes	4,778
Fair value of conversion feature of convertible senior unsecured notes	94	8,453	9,750
Special dividend - common stock issued		10,079
Redemption of OP Units for common stock		2,939
Special dividend - special voting preferred stock and OP Units issued		2,478
Cumulative-effect adjustment
Supplemental schedule of non-cash investing and financing activities:
Cumulative-effect adjustment (Note 2)	28,607	28,607
8.25% Series A preferred stock
Supplemental schedule of non-cash investing and financing activities:
Distributions accrued on preferred stock	267	267	267
7.75% Series B preferred stock
Supplemental schedule of non-cash investing and financing activities:
Distributions accrued on preferred stock	203	203	203
8.50% Series C preferred stock
Supplemental schedule of non-cash investing and financing activities:
Distributions accrued on preferred stock	$ 159	$ 159	$ 159