Net Interest Income - Components of Interest Income and Interest Expense (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2021	Jun. 30, 2020	Jun. 30, 2021	Jun. 30, 2020
Interest income	$ 52,186	$ 47,970	$ 102,225	$ 258,583
Derivatives	0	0	0	868
Interest expense	20,711	19,444	40,410	182,976
Total net interest income	31,475	28,526	61,815	75,607
Convertible Notes
Interest expense	2,788	2,739	5,572	5,474
Unsecured Debt
Senior unsecured notes	1,136	0	1,136	0
Subordinated debentures
Subordinated debentures	459	582	916	1,231
Residential loans
Interest income	20,155	17,471	39,816	39,153
Consolidated SLST | VIE, Primary Beneficiary
Interest income	10,479	11,522	20,797	23,646
Interest expense	7,151	8,158	14,254	16,693
Residential loans held in securitization trusts
Interest income	9,933	427	17,901	921
Total residential loans
Interest income	40,567	29,420	78,514	63,720
Preferred equity and mezzanine loan investments
Interest income	4,130	5,202	8,531	10,575
Consolidated K-Series | VIE, Primary Beneficiary
Interest income	0	0	0	151,841
Interest expense	0	0	0	129,762
Total multi-family loans
Interest income	4,130	5,202	8,531	162,416
Investment securities available for sale, at fair value
Investment securities available for sale	7,475	13,269	15,150	32,026
Other
Other	14	79	30	421
Other	429	0	741	0
Repurchase agreements
Interest expense	3,733	7,366	7,773	28,112
Residential loan securitizations
Interest expense	5,015	130	9,735	367
Non-Agency RMBS Re-Securitization
Interest expense	0	469	283	469
Total collateralized debt obligations
Interest expense	$ 12,166	$ 8,757	$ 24,272	$ 147,291