Condensed Quarterly Financial Data (Unaudited) (Details) - USD ($) $ / shares in Units, $ in Thousands	3 Months Ended													12 Months Ended
	Dec. 31, 2020		Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2020		Dec. 31, 2019		Dec. 31, 2018
Text Block [Abstract]
Interest income	$ 39,067		$ 43,075	$ 39,281	$ 52,108	$ 45,353	$ 39,985	$ 38,547	$ 36,016	$ 35,694	$ 35,300	$ 31,941	$ 28,092	$ 173,531		$ 159,901		$ 131,027	[1]
Interest expense	(11,952)		(12,937)	(14,686)	(22,090)	(21,205)	(19,954)	(19,702)	(17,618)	(16,083)	(15,678)	(13,383)	(11,562)	(61,665)		(78,479)		(56,707)	[1]
Interest Income (Expense), Net	27,115		30,138	24,595	30,018	24,148	20,031	18,845	18,398					111,866		81,422
Realized gains (losses) on securities and loans, financial derivatives, and real estate owned, net	(9,551)		(192)	(27,927)	204	(7,266)	(4,827)	(12,327)	(16,950)
Unrealized gains (losses) on securities and loans, financial derivatives, and real estate owned, net	42,547		24,032	52,057	(144,079)	6,139	7,970	13,300	20,452
Other, net	(795)		(2,747)	(435)	1,679	1,001	539	1,808	2,002					(2,298)		5,350
Noninterest Income	32,201		21,093	23,695	(142,196)	(126)	3,682	2,781	5,504					(65,207)		11,841
Base management fee to affiliate (Net of fee rebates)	3,178	[2]	2,981	2,906	2,443	2,663	1,942	1,661	1,722	1,744	1,830	2,021	1,978	11,508	[2]	7,988	[2]	7,573	[3]
Incentive Fee Expense	0		0	0	0	116	0	0	0	0	424	291	0	0		116		715
Servicing expense	1,736		2,379	2,493	2,531	2,055	1,940	2,244	2,393					9,139		8,632		7,715
Debt issuance costs related to Other secured borrowings, at fair value	1,819		0	2,075	0	1,865	0	1,671	0					3,894		3,536		1,647	[4]
Other	1,782		1,199	707	1,423	1,941	1,347	1,238	1,083					5,111		5,609		7,592
Professional fees	1,186		1,209	1,333	1,277	1,021	698	1,178	1,956					5,005		4,853		3,902
Compensation expense	962		1,085	941	788	962	712	903	1,072					3,776		3,649		2,233
Other expenses	1,531		1,625	1,497	1,752	1,160	1,156	1,053	985					6,405		4,354		1,898
Operating Expenses	12,194		10,478	11,952	10,214	11,783	7,795	9,948	9,211	26,637	24,668	22,044	18,566	44,838		38,737		91,916
Income (Loss) from Continuing Operations before Equity Method Investments, Income Taxes, Noncontrolling Interest	47,122		40,753	36,338	(122,392)	12,239	15,918	11,678	14,691					1,821		54,526
Income Tax Expense (Benefit)	7,888		2,494	1,542	(547)	1,180	2	376	0					11,377		1,558
Earnings (losses) from investments in unconsolidated entities	27,344		11,443	5,643	(6,497)	3,262	2,796	2,354	1,797					37,933		10,209
Net Income (Loss), Including Portion Attributable to Noncontrolling Interest, Total	66,578		49,702	40,439	(128,342)	14,321	18,712	13,656	16,488	(1,070)	7,474	22,184	21,324	28,377		63,177		49,911
Net income (loss) attributable to non-controlling interests	1,475		1,559	1,220	(885)	1,733	1,419	1,012	1,080	1,147	813	991	285	3,369		5,244		3,235
Dividends on preferred stock	1,941		1,940	1,941	1,941	1,466	0	0	0					7,763		1,466
Net income (loss) attributable to common stockholders	$ 63,162		$ 46,203	$ 37,278	$ (129,398)	$ 11,122	$ 17,293	$ 12,644	$ 15,408	$ (2,217)	$ 6,661	$ 21,193	$ 21,039	$ 17,245		$ 56,467		$ 46,676
Basic and Diluted (USD per share)	$ 1.44		$ 1.06	$ 0.85	$ (3.04)	$ 0.31	$ 0.53	$ 0.43	$ 0.52	$ (0.07)	$ 0.22	$ 0.69	$ 0.67	$ 0.39		$ 1.76		$ 1.52
Other income										$ 1,157	$ 1,046	$ 1,094	$ 716					$ 4,014
Total investment income										36,851	36,346	33,035	28,808					135,041
Other investment related expenses										4,201	4,384	3,771	2,952
Other operating expenses										4,609	2,352	2,578	2,074
Net investment income										10,214	11,678	10,991	10,242					43,125
Realized gain or loss on investments, financial derivatives, and foreign currency transactions (excludes FX on cash)										9,578	10,102	(1,343)	13,051					25,368
Total Net Unrealized Gain Loss On Investments, Other Secured Borrowings, Financial Derivatives, and Foreign Currency Translation (excluding FX on cash)										(20,862)	(14,306)	12,536	(1,969)					26,318
Investment Company, Realized and Unrealized Gain (Loss) on Investment and Foreign Currency										(11,284)	(4,204)	11,193	11,082					6,786
Net Income (Loss), Including Portion Attributable to Noncontrolling Interest	$ 66,578		$ 49,702	$ 40,439	$ (128,342)	$ 14,321	$ 18,712	$ 13,656	$ 16,488	(1,070)	7,474	22,184	21,324	$ 28,377		$ 63,177		$ 49,911
Management Fee Expense, Rebates										$ 400	$ 400	$ 300	$ 300	$ 1,051		$ 1,967	[2]

[1] Includes interest income and interest expense of a consolidated securitization trust of $6.0 million and $3.6 million, respectively, for the year ended December 31, 2018. See Note 6 for further details on the Company's consolidated securitization trust.

[2] See Note 13 for further details on management fee rebates.

[3] See Note 9 for further details on management fee rebates.

[4] Related to non-qualified mortgage securitization transactions. See Note 6 for further details.