FAIR VALUE OF FINANCIAL INSTRUMENTS - Estimated Fair Values of Financial Instruments (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2023 USD ($)	Dec. 31, 2022 USD ($)	Dec. 31, 2021 USD ($)	Dec. 31, 2020 USD ($)
Assets:
Fair Value	$ 3,192,150	$ 3,922,131
Liabilities:
Fair Value	3,683,621	3,994,458
Allowance for credit losses	$ (43,165)	$ (20,755)	$ (31,752)	$ (41,507)
Total mortgage loan receivables held for investment, net, at amortized cost
Liabilities:
Period of short interest rate reset risk	30 days	30 days
CLO debt
Liabilities:
Period of short interest rate reset risk		30 days
Recurring
Assets:
Fair Value	$ 477,743	$ 580,130
Recurring | Agency securities
Assets:
Principal Amount	22	36
Fair Value	21	35
Recurring | CMBS | Internal model, third-party inputs
Assets:
Principal Amount	439,679	562,839
Amortized Cost Basis/Purchase Price	439,052	562,246
Fair Value	$ 424,890	$ 541,333
Liabilities:
Financial instruments, measurement input	0.0683	0.0532
Weighted average remaining maturity/duration	2 years	1 year 21 days
Recurring | CMBS interest-only | Internal model, third-party inputs
Assets:
Principal Amount	$ 876,555	$ 1,026,195
Amortized Cost Basis/Purchase Price	6,453	10,498
Fair Value	$ 6,569	$ 10,443
Liabilities:
Financial instruments, measurement input	0.0661	0.0365
Weighted average remaining maturity/duration	1 year 25 days	1 year 5 months 12 days
Recurring | GNMA interest-only | Internal model, third-party inputs
Assets:
Principal Amount	$ 37,053	$ 45,369
Amortized Cost Basis/Purchase Price	214	285
Fair Value	$ 213	$ 281
Liabilities:
Financial instruments, measurement input	0.0612	0.0423
Weighted average remaining maturity/duration	3 years 7 months 6 days	3 years 3 months 18 days
Recurring | Agency securities | Internal model, third-party inputs
Assets:
Principal Amount	$ 22	$ 36
Amortized Cost Basis/Purchase Price	22	36
Fair Value	$ 21	$ 35
Liabilities:
Financial instruments, measurement input	0.0270	0.0270
Weighted average remaining maturity/duration	1 year 18 days	1 year 6 months 14 days
Recurring | U.S. Treasury securities | Internal model, third-party inputs
Assets:
Principal Amount	$ 54,031	$ 36,000
Amortized Cost Basis/Purchase Price	53,648	35,328
Fair Value	$ 53,716	$ 35,328
Liabilities:
Financial instruments, measurement input	0.0541	0.0417
Weighted average remaining maturity/duration	25 days	7 months 6 days
Recurring | Equity securities
Assets:
Amortized Cost Basis/Purchase Price	$ 160	$ 160
Fair Value	144	118
Recurring | Total mortgage loan receivables held for investment, net, at amortized cost | Discounted Cash Flow
Assets:
Principal Amount	3,164,226	3,907,295
Amortized Cost Basis/Purchase Price	3,155,089	3,885,746
Fair Value	$ 3,150,843	$ 3,875,708
Liabilities:
Financial instruments, measurement input	0.0965	0.0885
Weighted average remaining maturity/duration	8 months 4 days	1 year 3 months 3 days
Allowance for credit losses	$ (43,200)	$ (20,800)
Recurring | Mortgage loan  receivables held for sale | Internal model, third-party inputs
Assets:
Principal Amount	31,350	31,350
Amortized Cost Basis/Purchase Price	26,868	27,391
Fair Value	$ 26,868	$ 27,391
Liabilities:
Financial instruments, measurement input	0.0457	0.0457
Weighted average remaining maturity/duration	8 years 2 months 8 days	9 years 2 months 8 days
Recurring | FHLB stock | FHLB stock
Assets:
Principal Amount	$ 5,175	$ 9,585
Amortized Cost Basis/Purchase Price	5,175	9,585
Fair Value	$ 5,175	$ 9,585
Liabilities:
Financial instruments, measurement input	0.0825	0.0475
Recurring | Nonhedge derivatives | Counterparty quotations
Assets:
Nonhedge derivative assets	$ 194,900	$ 204,700
Amortized Cost Basis/Purchase Price	1,454	2,038
Fair Value	$ 1,454	$ 2,038
Liabilities:
Weighted average remaining maturity/duration	5 months 23 days	1 year 6 months 7 days
Recurring | Repurchase agreements - short-term | Cost plus Accrued Interest
Liabilities:
Principal Amount	$ 337,631	$ 481,465
Amortized Cost Basis/Purchase Price	337,631	481,465
Fair Value	$ 337,631	$ 481,465
Financial instruments, measurement input	0.0757	0.0404
Weighted average remaining maturity/duration	5 months 23 days	4 months 13 days
Recurring | Repurchase agreements - long-term | Discounted Cash Flow
Liabilities:
Principal Amount	$ 268,976	$ 366,398
Amortized Cost Basis/Purchase Price	268,976	366,398
Fair Value	$ 268,976	$ 366,398
Financial instruments, measurement input	0.0735	0.0406
Weighted average remaining maturity/duration	1 year 8 months 26 days	2 years 6 months 21 days
Recurring | Mortgage loan financing | Discounted Cash Flow
Liabilities:
Principal Amount	$ 437,384	$ 497,454
Amortized Cost Basis/Purchase Price	437,759	497,991
Fair Value	$ 425,992	$ 477,101
Financial instruments, measurement input	0.0587	0.0551
Weighted average remaining maturity/duration	2 years 7 months 20 days	3 years 4 months 9 days
Recurring | CLO debt | Discounted Cash Flow
Liabilities:
Principal Amount	$ 1,062,777	$ 1,064,365
Amortized Cost Basis/Purchase Price	1,060,719	1,058,462
Fair Value	$ 1,060,719	$ 1,058,462
Financial instruments, measurement input	0.0708	0.0635
Weighted average remaining maturity/duration	1 year 10 months 20 days	15 years 11 months 1 day
Recurring | Borrowings from the FHLB | Discounted Cash Flow
Liabilities:
Principal Amount	$ 115,000	$ 213,000
Amortized Cost Basis/Purchase Price	115,000	213,000
Fair Value	$ 115,000	$ 213,055
Financial instruments, measurement input	0.0582	0.0161
Weighted average remaining maturity/duration	6 months 25 days	1 year 3 months
Recurring | Senior unsecured notes | Internal model, third-party inputs
Liabilities:
Principal Amount	$ 1,575,614	$ 1,643,794
Amortized Cost Basis/Purchase Price	1,563,861	1,628,382
Fair Value	$ 1,475,303	$ 1,397,977
Financial instruments, measurement input	0.0466	0.0466
Weighted average remaining maturity/duration	3 years 9 months 7 days	4 years 9 months