Related Party Transactions (Details) - USD ($) $ in Thousands		3 Months Ended																12 Months Ended
		Dec. 31, 2018		Sep. 30, 2018		Jun. 30, 2018		Mar. 31, 2018		Dec. 31, 2017		Sep. 30, 2017		Jun. 30, 2017	[2]	Mar. 31, 2017	[2]	Dec. 31, 2018		Dec. 31, 2017
Related party transactions [Line Items]
Annual base management fee percentage																		1.50%
Management fee description																		The Operating Partnership pays the Manager 1.50% per annum of total equity of the Operating Partnership calculated in accordance with U.S. GAAP as of the end of each fiscal quarter (before deductions for base management fees and incentive fees payable with respect to such fiscal quarter), provided that total equity is adjusted to exclude one-time events pursuant to changes in U.S. GAAP, as well as non-cash charges after discussion between the Manager and the Company's independent directors, and approval by a majority of the Company's independent directors in the case of non-cash charges.
Base management fee to affiliate		$ 1,744	[1]	$ 1,830	[1]	$ 2,021	[1]	$ 1,978	[1]	$ 2,113	[2]	$ 2,161	[2]	$ 2,372		$ 2,410		$ 7,573	[3]	$ 9,056	[3]
Incentive fee rate																		25.00%
Incentive fees arrangement																		The Manager is entitled to receive a quarterly incentive fee equal to the positive excess, if any, of (i) the product of (A) 25% and (B) the excess of (1) Adjusted Net Income (described below) for the Incentive Calculation Period (which means such fiscal quarter and the immediately preceding three fiscal quarters) over (2) the sum of the Hurdle Amounts (described below) for the Incentive Calculation Period, over (ii) the sum of the incentive fees already paid or payable for each fiscal quarter in the Incentive Calculation Period preceding such fiscal quarter. For purposes of calculating the incentive fee, "Adjusted Net Income" for the Incentive Calculation Period means the net increase in equity from operations of the Operating Partnership, after all base management fees but before any incentive fees for such period, and excluding any non-cash equity compensation expenses for such period, as reduced by any Loss Carryforward (as described below) as of the end of the fiscal quarter preceding the Incentive Calculation Period. For purposes of calculating the incentive fee, the "Loss Carryforward" as of the end of any fiscal quarter is calculated by determining the excess, if any, of (1) the Loss Carryforward as of the end of the immediately preceding fiscal quarter over (2) the Company's net increase in equity from operations (expressed as a positive number) or net decrease in equity from operations (expressed as a negative number) of the Operating Partnership for such fiscal quarter. As of December 31, 2018, there was a Loss Carryforward of $2.1 million. For purposes of calculating the incentive fee, the "Hurdle Amount" means, with respect to any fiscal quarter, the product of (i) one-fourth of the greater of (A) 9% and (B) 3% plus the 10-year U.S. Treasury rate for such fiscal quarter, (ii) the sum of (A) the weighted average gross proceeds per share of all common share and OP Unit issuances since inception of the Company and up to the end of such fiscal quarter, with each issuance weighted by both the number of shares and OP Units issued in such issuance and the number of days that such issued shares and OP Units were outstanding during such fiscal quarter, using a first-in first-out basis of accounting (i.e. attributing any share and OP Unit repurchases to the earliest issuances first) and (B) the result obtained by dividing (I) retained earnings attributable to common shares and OP Units at the beginning of such fiscal quarter by (II) the average number of common shares and OP Units outstanding for each day during such fiscal quarter, (iii) the sum of (x) the average number of common shares and LTIP Units outstanding for each day during such fiscal quarter, and (y) the average number of OP Units and OP LTIP Units outstanding for each day during such fiscal quarter. For purposes of determining the Hurdle Amount, issuances of common shares, OP LTIP Units, and OP Units (a) as equity incentive awards, (b) to the Manager as part of its base management fee or incentive fee and (c) to the Manager or any of its affiliates in privately negotiated transactions, are excluded from the calculation. The payment of the incentive fee will be in a combination of common shares and cash, provided that at least 10% of any quarterly payment will be made in common shares.
Termination Fee, Description																		The Management Agreement requires the Company to pay a termination fee to the Manager in the event of (1) the Company's termination or non-renewal of the Management Agreement without cause or (2) the Company's termination of the Management Agreement based on unsatisfactory performance by the Manager that is materially detrimental to the Company or (3) the Manager's termination of the Management Agreement upon a default by the Company in the performance of any material term of the Management Agreement. Such termination fee will be equal to the amount of three times the sum of (i) the average annual Quarterly Base Management Fee Amounts paid or payable with respect to the two 12-month periods ending on the last day of the latest fiscal quarter completed on or prior to the date of the notice of termination or non-renewal and (ii) the average annual Quarterly Incentive Fee Amounts paid or payable with respect to the two 12-month periods ending on the last day of the latest fiscal quarter completed on or prior to the date of the notice of termination or non-renewal.
Incentive fee hurdle rate fixed																		9.00%
Incentive fee hurdle rate floating																		3.00%
Minimum percentage of incentive fee to be paid in share																		10.00%
Incentive fee to affiliate		0		424		291		0										$ 715		0
Expense Reimbursement Period																		60 days
Expense Reimbursement - Manager																		$ 6,500		6,400
Investments Fair Value		2,939,311								2,071,707								2,939,311		2,071,707
Non-controlling interests		31,337								20,862								31,337		20,862
Other secured borrowings		114,100								57,909								114,100		57,909
Reverse repurchase agreements	[4]	1,498,849								1,209,315								1,498,849		1,209,315
Fair value repurchase agreements	[4]	61,274								155,949								61,274		155,949
Investments held with dealers as collateral		1,790,000								1,410,000								1,790,000		1,410,000
Management fee rebate	[3]	400		$ 400		$ 300		$ 300		200		$ 200						1,380		332
Other assets		15,536								43,770								15,536		43,770
Face amount of investments purchased from related party																		2,900
Commercial mortgage loans and REO [Member]
Related party transactions [Line Items]
Investments Fair Value	[5],[6],[7],[8]	25,300								27,900								25,300		27,900
Reverse repurchase agreements		77,000								56,900								77,000		56,900
Corporate Equity Investments [Member]
Related party transactions [Line Items]
Investments Fair Value		43,793	[9],[10]							37,465	[7],[8]							43,793	[9],[10]	37,465	[7],[8]
Secured Notes [Member]
Related party transactions [Line Items]
Investments Fair Value	[9],[10],[11]	10,917																10,917
Corporate Debt [Member]
Related party transactions [Line Items]
Investments Fair Value		22,392	[9],[10]							75,193	[7],[8]							22,392	[9],[10]	75,193	[7],[8]
Mortgage Originator [Member]
Related party transactions [Line Items]
Warehouse facility, monetary amount		$ 5,000																$ 5,000
Line of Credit Facility, Interest Rate at Period End		15.00%																15.00%
Advances made under warehouse facility		$ 0																$ 0
Related Party-Consumer Loans Titled in Name of Related Party [Member]
Related party transactions [Line Items]
Fair value of loans held in related party trust		21,900								11,700								21,900		11,700
Purchasing Entity [Member]
Related party transactions [Line Items]
Fair value of loans held in related party trust		181,500								114,500								181,500		114,500
Purchase commitment principal balance																		166,300
Estimated remaining contingent purchase obligations		263,500																263,500
Residential mortgage loans and REO [Member]
Related party transactions [Line Items]
Fair value of loans held in related party trust		498,100								183,100								498,100		183,100
Non-controlling interest of joint venture partner [Member] | Unrelated third party joint venture interest [Member] | Commercial mortgage loans and REO [Member]
Related party transactions [Line Items]
Non-controlling interests		1,400								1,800								1,400		1,800
Non-controlling interest of joint venture partner [Member] | Related party joint venture interest [Member] | Commercial mortgage loans and REO [Member]
Related party transactions [Line Items]
Non-controlling interests		4,100								5,300								4,100		5,300
Participation in multi-borrower financing facility [Member]
Related party transactions [Line Items]
Outstanding debt of related party		149,000								106,600								149,000		106,600
Due from Related Parties										23,400										23,400
Participation in Multi-Seller Consumer Loan Securitization [Member] | Repurchase Agreements [Member] | Acquiror Repurchase Agreement [Member]
Related party transactions [Line Items]
Fair value repurchase agreements										5,700										5,700
Investments held with dealers as collateral	[9],[11],[12]									9,400										9,400
Loan receivable from affiliated entity related to warehouse facility [Member]
Related party transactions [Line Items]
Other assets		11,600								$ 16,900								11,600		$ 16,900
Investment purchased from related party [Member] | Corporate Debt [Member]
Related party transactions [Line Items]
Investments Fair Value		1,600																1,600
Mortgage-related Commercial [Member] | Corporate Equity Investments [Member]
Related party transactions [Line Items]
Investments Fair Value		$ 1,100																$ 1,100
Ownership percentage of limited liability company held as investment		15.00%																15.00%

[1] Net of management fee rebate of $0.3 million, $0.3 million, $0.4 million, and $0.4 million, for the each of the three-month periods ended March 31, 2018, June 30, 2018, September 30, 2018, and December 31, 2018, respectively. See Note 9 for further details on management fee rebates.

[2] Net of management fee rebate of $0.2 million for the each of the three-month periods ended September 30, 2017 and December 31, 2017, respectively. See Note 9 for further details on management fee rebates.

[3] See Note 9 for further details on management fee rebates.

[4] In the Company's Consolidated Statement of Assets, Liabilities, and Equity, all balances associated with repurchase agreements, reverse repurchase agreements, and financial derivatives are presented on a gross basis.

[5] Includes non-performing commercial mortgage loans in the amount of $23.9 million whereby principal and/or interest is past due and a maturity date is not applicable.

[6] Loans and real estate owned are beneficially owned by the Company through participation certificates in the various trusts that hold such investments. See Note 9 to the Notes to Consolidated Financial Statements.

[7] See Note 2 and Note 3 in Notes to Consolidated Financial Statements.

[8] The table below shows the ratings on the Company's long investments from Moody's, Standard and Poor's, or Fitch, as well as the Company's long investments that were unrated but guaranteed by the Federal National Mortgage Association, the Federal Home Loan Mortgage Corporation, or the Government National Mortgage Association. Ratings tend to be a lagging credit indicator; as a result, the credit quality of the Company's long investment holdings may be lower than the credit quality implied based on the ratings listed below. In situations where an investment has a split rating, the lowest provided rating is used. The ratings descriptions include ratings qualified with a "+," "-," "1," "2," or "3."Rating Description Percent of EquityUnrated but Agency-Guaranteed 160.32%A/A/A 0.81%Baa/BBB/BBB 2.62%Ba/BB/BB or below 68.03%Unrated 101.85%

[9] See Note 2 and Note 3 in Notes to Consolidated Financial Statements.

[10] The table below shows the Company's long investment ratings from Moody's, Standard and Poor's, or Fitch, as well as the Company's long investments that were unrated but guaranteed by the Federal National Mortgage Association, the Federal Home Loan Mortgage Corporation, or the Government National Mortgage Association. Ratings tend to be a lagging credit indicator; as a result, the credit quality of the Company's long investment holdings may be lower than the credit quality implied based on the ratings listed below. In situations where an investment has a split rating, the lowest provided rating is used. The ratings descriptions include ratings qualified with a "+," "-," "1," "2," or "3."Rating Description Percent of EquityUnrated but Agency-Guaranteed 243.66%Aaa/AAA/AAA 0.01%Aa/AA/AA 0.63%A/A/A 4.73%Baa/BBB/BBB 1.84%Ba/BB/BB or below 46.34%Unrated 196.65%

[11] Represents the Company's investment in a related party. See Note 9 to the Notes to Consolidated Financial Statements

[12] Includes $314.2 million of non-qualified mortgage loans that have been securitized and are held in a consolidated securitization trusts. See Note 6 to the Notes to Consolidated Financial Statements.