Condensed Consolidated Statement of Cash Flows - USD ($) $ in Thousands		9 Months Ended
		Sep. 30, 2020	Sep. 30, 2019
Statement of Cash Flows [Abstract]
Net Income (Loss)		$ (38,201)	$ 48,856
Reconciliation of the net increase (decrease) in equity resulting from operations to net cash provided by (used in) operating activities:
Realized (gains) losses on securities and loans, net		2,335	3,460
Realized (gains) losses on financial derivatives, net		25,702	31,850
Realized (gains) losses on real estate owned, net		(121)	(1,205)
Unrealized (gains) losses on securities and loans, net		65,418	(51,395)
Unrealized (gains) losses on financial derivatives, net		2,109	9,136
Unrealized (gains) losses on real estate owned, net		462	535
Share-based long term incentive plan unit expense		500	300
Debt issuance costs related to Other secured borrowings, at fair value(1)		0	(1,034)
(Earnings) losses from investments in unconsolidated entities		(10,590)	(6,947)
Increase (decrease) in liabilities:
Net cash provided by (used in) operating activities		82,480	56,119
Cash Flows from Investing Activities:
Purchase of securities		(1,189,684)	(1,890,456)
Purchase of loans		(548,104)	(701,569)
Capital improvements of real estate owned		(153)	(240)
Proceeds from disposition of securities		1,830,612	1,353,755
Proceeds from disposition of loans		8,006	28,344
Contributions to investments in unconsolidated entities		(19,711)	(39,369)
Distributions from investments in unconsolidated entities		17,181	43,587
Proceeds from disposition of real estate owned		7,801	5,579
Proceeds from principal payments of securities		284,242	175,804
Proceeds from principal payments of loans		337,748	207,413
Proceeds from investments sold short		264,267	508,074
Repurchase of securities sold short		(292,105)	(549,782)
Payments on financial derivatives		(99,800)	(79,558)
Proceeds from financial derivatives		68,546	45,368
Payments made on reverse repurchase agreements		(6,659,967)	(4,171,036)
Proceeds from reverse repurchase agreements		6,686,565	4,343,602
Due from brokers, net		12,376	10,160
Due to brokers, net		5,641	(2,995)
Net cash provided by (used in) investing activities		713,461	(713,319)
Cash flows provided by (used in) financing activities:
Net proceeds from issuance of common stock	[1]	95,537	70,035
Offering costs paid		(253)	(462)
Repurchase of common stock		(3,056)	(782)
Dividends paid		(50,642)	(39,226)
Contributions from non-controlling interests		9,134	16,876
Distributions to non-controlling interests		(7,961)	(17,250)
Proceeds from issuance of securitized debt		45,963	28,490
Proceeds from issuance of Other secured borrowings		205,357	70,988
Principal payment on securitized debt		(59,893)	(51,439)
Borrowings under repurchase agreements		3,445,412	6,110,390
Repayments of repurchase agreements		(4,423,440)	(5,540,301)
Due from brokers, net		2,620	(3,826)
Due to brokers, net		(238)	3,086
Net cash provided by (used in) financing activities		(741,460)	645,545
Net Increase (Decrease) in Cash, Cash Equivalents, and Restricted Cash		54,481	(11,655)
Cash, Cash Equivalents, and Restricted Cash, Beginning of Period		72,477	45,081
Cash, Cash Equivalents, and Restricted Cash, End of Period		126,958	33,426
Supplemental disclosure of cash flow information:
Interest paid		54,958	58,298
Dividends payable		5,299	4,833
Contribution from non-controlling interest (non-cash)		2,340	0
Distributions to non-controlling interests		(2,340)	0
Income tax paid		505	142
Share-based long term incentive plan unit awards (non-cash)		500	300
Transfers from mortgage loans to real estate owned (non-cash)		2,198	18,314
Transfers from mortgage loans to investments in non-consolidated entities (non-cash)		10,833	0
Purchase of investments (non-cash)		0	(2,975)
Purchase of Loans Non Cash		6,270	0
Proceeds from issuance of Other secured borrowings (non-cash)		6,270	0
Proceeds from principal payments of investments (non-cash)		138,608	78,887
Repayments of repurchase agreement (non-cash)		(27,288)	(148,158)
Principal payments on Other secured borrowings, at fair value (non-cash)		(138,608)	(78,887)
Proceeds received from Other secured borrowings, at fair value (non-cash)		27,735	148,547
Repayment of senior notes (non-cash)		0	(86,000)
Issuance of senior notes (non-cash)		$ 0	$ 86,000

[1] Net of underwriters' discounts.