Condensed Consolidated Statement of Cash Flows - USD ($) $ in Thousands		3 Months Ended
		Mar. 31, 2020	Mar. 31, 2019
Statement of Cash Flows [Abstract]
Net Income (Loss)		$ (128,342)	$ 16,488
Reconciliation of the net increase (decrease) in equity resulting from operations to net cash provided by (used in) operating activities:
Realized (gains) losses on securities and loans, net		(12,260)	5,322
Realized (gains) losses on financial derivatives, net		12,406	11,570
Realized (gains) losses on real estate owned, net		(350)	58
Unrealized (gains) losses on securities and loans, net		133,738	(26,388)
Unrealized (gains) losses on financial derivatives, net		9,984	5,689
Unrealized (gains) losses on real estate owned, net		357	247
Share-based long term incentive plan unit expense		200	100
(Earnings) losses from investments in unconsolidated entities		6,497	(1,797)
Increase (decrease) in liabilities:
Net cash provided by (used in) operating activities		28,867	20,992
Cash Flows from Investing Activities:
Purchase of securities		(827,252)	(617,445)
Purchase of loans		(273,571)	(260,716)
Capital improvements of real estate owned		0	(240)
Proceeds from disposition of securities		1,241,449	682,337
Proceeds from disposition of loans		38	10,296
Contributions to investments in unconsolidated entities		(3,034)	(13,245)
Distributions from investments in unconsolidated entities		13,824	27,585
Proceeds from disposition of real estate owned		6,946	9
Proceeds from principal payments of securities		135,093	36,414
Proceeds from principal payments of loans		154,441	87,481
Proceeds from investments sold short		201,998	278,033
Repurchase of securities sold short		(265,932)	(329,382)
Payments on financial derivatives		(48,081)	(32,285)
Proceeds from financial derivatives		26,877	22,161
Payments made on reverse repurchase agreements		(5,880,200)	(1,536,791)
Proceeds from reverse repurchase agreements		5,940,600	1,572,683
Due from brokers, net		(3,029)	11,026
Due to brokers, net		8,287	(2,476)
Net cash provided by (used in) investing activities		428,454	(64,555)
Cash flows provided by (used in) financing activities:
Net proceeds from issuance of common stock	[1]	95,537	0
Offering costs paid		(253)	0
Repurchase of common stock		(3,035)	(782)
Dividends paid		(21,024)	(12,497)
Contributions from non-controlling interests		3,487	2,512
Distributions to non-controlling interests		(4,798)	(4,306)
Proceeds from issuance of securitized debt		31,913	16,680
Principal payment on securitized debt		(10,662)	(13,465)
Borrowings under repurchase agreements		1,856,666	1,507,161
Repayments of repurchase agreements		(2,267,741)	(1,443,871)
Due from brokers, net		(73,544)	3,707
Due to brokers, net		571	(606)
Net cash provided by (used in) financing activities		(392,883)	54,533
Net Increase (Decrease) in Cash, Cash Equivalents, and Restricted Cash		64,438	10,970
Cash, Cash Equivalents, and Restricted Cash, Beginning of Period		72,477	45,081
Cash, Cash Equivalents, and Restricted Cash, End of Period		136,915	56,051
Supplemental disclosure of cash flow information:
Interest paid		24,126	19,782
Dividends payable		7,952	4,267
Share-based long term incentive plan unit awards (non-cash)		200	100
Transfers from mortgage loans to real estate owned (non-cash)		1,422	299
Transfers from mortgage loans to investments in non-consolidated entities (non-cash)		10,833	0
Purchase of loans (non-cash)		6,270	0
Proceeds from issuance of Other secured borrowings (non-cash)		6,270	0
Proceeds from principal payments of investments (non-cash)		44,704	15,767
Principal payments on Other secured borrowings, at fair value (non-cash)		(44,704)	(15,767)
Repayment of senior notes (non-cash)		0	(86,000)
Issuance of senior notes (non-cash)		$ 0	$ 86,000

[1] Net of underwriters' discounts.