Segment Information - Statements of Income (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2024 USD ($) segment	Mar. 31, 2023 USD ($)
Segment Information
Interest income	$ 321,292	$ 327,947
Interest expense	217,676	219,373
Net interest income	103,616	108,574
Other revenue:
Gain on sales, including fee-based services, net	16,666	14,589
Mortgage servicing rights	10,199	18,458
Servicing revenue	48,157	44,981
Amortization of MSRs	(16,631)	(15,416)
Property operating income	1,570	1,381
Gain (loss) on derivative instruments, net	(5,257)	4,223
Other income, net	2,333	4,882
Total other revenue	57,037	73,098
Other expenses:
Employee compensation and benefits	47,694	42,399
Selling and administrative	13,933	13,623
Property operating expenses	1,678	1,383
Depreciation and amortization	2,571	2,624
Provision for loss sharing (net of recoveries)	273	3,177
Provision for credit losses (net of recoveries)	19,118	22,517
Total other expenses	85,267	85,723
Income before income from equity affiliates and income taxes	75,386	95,949
Income from equity affiliates	1,418	14,326
Provision for income taxes	(3,592)	(8,029)
Net income	73,212	102,246
Preferred stock dividends	10,342	10,342
Net income attributable to noncontrolling interest	4,997	7,585
Net income attributable to common stockholders	$ 57,873	84,319
Reporting segments | segment	2
Operating segments | Structured Business
Segment Information
Interest income	$ 307,888	317,376
Interest expense	212,600	214,894
Net interest income	95,288	102,482
Other revenue:
Gain on sales, including fee-based services, net	0	0
Mortgage servicing rights	0	0
Servicing revenue	0	0
Amortization of MSRs	0	0
Property operating income	1,570	1,381
Gain (loss) on derivative instruments, net	0	0
Other income, net	2,300	1,908
Total other revenue	3,870	3,289
Other expenses:
Employee compensation and benefits	18,547	15,641
Selling and administrative	6,796	6,711
Property operating expenses	1,678	1,383
Depreciation and amortization	1,398	1,451
Provision for loss sharing (net of recoveries)	0	0
Provision for credit losses (net of recoveries)	17,777	20,645
Total other expenses	46,196	45,831
Income before income from equity affiliates and income taxes	52,962	59,940
Income from equity affiliates	1,418	14,326
Provision for income taxes	(81)	429
Net income	54,299	74,695
Preferred stock dividends	10,342	10,342
Net income attributable to noncontrolling interest	0	0
Net income attributable to common stockholders	43,957	64,353
Operating segments | Agency Business
Segment Information
Interest income	13,404	10,571
Interest expense	5,076	4,479
Net interest income	8,328	6,092
Other revenue:
Gain on sales, including fee-based services, net	16,666	14,589
Mortgage servicing rights	10,199	18,458
Servicing revenue	48,157	44,981
Amortization of MSRs	(16,631)	(15,416)
Property operating income	0	0
Gain (loss) on derivative instruments, net	(5,257)	4,223
Other income, net	33	2,974
Total other revenue	53,167	69,809
Other expenses:
Employee compensation and benefits	29,147	26,758
Selling and administrative	7,137	6,912
Property operating expenses	0	0
Depreciation and amortization	1,173	1,173
Provision for loss sharing (net of recoveries)	273	3,177
Provision for credit losses (net of recoveries)	1,341	1,872
Total other expenses	39,071	39,892
Income before income from equity affiliates and income taxes	22,424	36,009
Income from equity affiliates	0	0
Provision for income taxes	(3,511)	(8,458)
Net income	18,913	27,551
Preferred stock dividends	0	0
Net income attributable to noncontrolling interest	0	0
Net income attributable to common stockholders	18,913	27,551
Other/Eliminations
Other expenses:
Net income attributable to noncontrolling interest	4,997	7,585
Net income attributable to common stockholders	$ (4,997)	$ (7,585)