Related Party Transactions (Details) - USD ($) $ in Thousands		3 Months Ended			6 Months Ended
		Jun. 30, 2018		Jun. 30, 2017	Jun. 30, 2018		Jun. 30, 2017	Dec. 31, 2017
Related party transactions [Line Items]
Annual base management fee percentage					1.50%
Management fee description					The Operating Partnership pays the Manager 1.50% per annum of total equity of the Operating Partnership calculated in accordance with U.S. GAAP as of the end of each fiscal quarter (before deductions for base management fees and incentive fees payable with respect to such fiscal quarter), provided that total equity is adjusted to exclude one-time events pursuant to changes in U.S. GAAP, as well as non-cash charges after discussion between the Manager and the Company's independent directors, and approval by a majority of the Company's independent directors in the case of non-cash charges.
Base management fee to affiliate	[1]	$ 2,021		$ 2,372	$ 4,000		$ 4,782
Incentive fee rate					25.00%
Incentive fees arrangement					The Manager is entitled to receive a quarterly incentive fee equal to the positive excess, if any, of (i) the product of (A) 25% and (B) the excess of (1) Adjusted Net Income (described below) for the Incentive Calculation Period (which means such fiscal quarter and the immediately preceding three fiscal quarters) over (2) the sum of the Hurdle Amounts (described below) for the Incentive Calculation Period, over (ii) the sum of the incentive fees already paid or payable for each fiscal quarter in the Incentive Calculation Period preceding such fiscal quarter. For purposes of calculating the incentive fee, "Adjusted Net Income" for the Incentive Calculation Period means the net increase in equity from operations of the Operating Partnership, after all base management fees but before any incentive fees for such period, and excluding any non-cash equity compensation expenses for such period, as reduced by any Loss Carryforward (as described below) as of the end of the fiscal quarter preceding the Incentive Calculation Period. For purposes of calculating the incentive fee, the "Loss Carryforward" as of the end of any fiscal quarter is calculated by determining the excess, if any, of (1) the Loss Carryforward as of the end of the immediately preceding fiscal quarter over (2) the Company's net increase in equity from operations (expressed as a positive number) or net decrease in equity from operations (expressed as a negative number) of the Operating Partnership for such fiscal quarter. As of June 30, 2018, there was no Loss Carryforward. For purposes of calculating the incentive fee, the "Hurdle Amount" means, with respect to any fiscal quarter, the product of (i) one-fourth of the greater of (A) 9% and (B) 3% plus the 10-year U.S. Treasury rate for such fiscal quarter, (ii) the sum of (A) the weighted average gross proceeds per share of all common share and OP Unit issuances since inception of the Company and up to the end of such fiscal quarter, with each issuance weighted by both the number of shares and OP Units issued in such issuance and the number of days that such issued shares and OP Units were outstanding during such fiscal quarter, using a first-in first-out basis of accounting (i.e. attributing any share and OP Unit repurchases to the earliest issuances first) and (B) the result obtained by dividing (I) retained earnings attributable to common shares and OP Units at the beginning of such fiscal quarter by (II) the average number of common shares and OP Units outstanding for each day during such fiscal quarter, (iii) the sum of the average number of common shares, LTIP Units, and OP Units outstanding for each day during such fiscal quarter. For purposes of determining the Hurdle Amount, issuances of common shares and OP Units (a) as equity incentive awards, (b) to the Manager as part of its base management fee or incentive fee and (c) to the Manager or any of its affiliates in privately negotiated transactions, are excluded from the calculation. The payment of the incentive fee will be in a combination of common shares and cash, provided that at least 10% of any quarterly payment will be made in common shares.
Termination Fee, Description					The Management Agreement requires the Company to pay a termination fee to the Manager in the event of (1) the Company's termination or non-renewal of the Management Agreement without cause or (2) the Company's termination of the Management Agreement based on unsatisfactory performance by the Manager that is materially detrimental to the Company (3) the Manager's termination of the Management Agreement upon a default by the Company in the performance of any material term of the Management Agreement. Such termination fee will be equal to the amount of three times the sum of (i) the average annual Quarterly Base Management Fee Amounts paid or payable with respect to the two 12-month periods ending on the last day of the latest fiscal quarter completed on or prior to the date of the notice of termination or non-renewal and (ii) the average annual Quarterly Incentive Fee Amounts paid or payable with respect to the two 12-month periods ending on the last day of the latest fiscal quarter completed on or prior to the date of the notice of termination or non-renewal.
Incentive fee hurdle rate fixed					9.00%
Incentive fee hurdle rate floating					3.00%
Minimum percentage of incentive fee to be paid in share					10.00%
Incentive fee		291		0	$ 291		0
Expense Reimbursement Period					60 days
Expense Reimbursement - Manager					$ 3,800		2,700
Investments Fair Value		2,625,471			2,625,471			$ 2,071,707
Non-controlling interests		13,708			13,708			20,862
Other secured borrowings		95,630			95,630			57,909
Reverse repurchase agreements	[2]	1,421,506			1,421,506			1,209,315
Fair value repurchase agreements	[2]	214,411			214,411			155,949
Investments held with dealers as collateral		1,690,000			1,690,000			1,410,000
Management fee rebate	[1]	252		$ 0	527		$ 0
Other assets		24,399			24,399			43,770
Commercial mortgage loans and REO [Member]
Related party transactions [Line Items]
Investments Fair Value	[3],[4],[5],[6]	20,300			20,300			27,900
Reverse repurchase agreements		66,600			66,600			56,900
Corporate Equity Investments [Member]
Related party transactions [Line Items]
Investments Fair Value		45,505	[7],[8]		45,505	[7],[8]		37,465	[5],[6]
Secured Notes [Member]
Related party transactions [Line Items]
Investments Fair Value	[7],[8],[9]	11,126			11,126
Mortgage Originator [Member]
Related party transactions [Line Items]
Warehouse facility, monetary amount		$ 5,000			$ 5,000
Line of Credit Facility, Interest Rate at Period End		15.00%			15.00%
Advances made under warehouse facility		$ 0			$ 0
Related Party-Consumer Loans Titled in Name of Related Party [Member]
Related party transactions [Line Items]
Fair value of loans held in related party trust		18,200			18,200			11,700
Purchasing Entity [Member]
Related party transactions [Line Items]
Fair value of loans held in related party trust		168,500			168,500			114,500
Purchase commitment principal balance					93,700
Estimated remaining contingent purchase obligations		111,600			111,600
Residential mortgage loans and REO [Member]
Related party transactions [Line Items]
Fair value of loans held in related party trust		294,400			294,400			183,100
Non-controlling interest of joint venture partner [Member] | Unrelated third party joint venture interest [Member] | Commercial mortgage loans and REO [Member]
Related party transactions [Line Items]
Non-controlling interests		1,000			1,000			1,800
Non-controlling interest of joint venture partner [Member] | Related party joint venture interest [Member] | Commercial mortgage loans and REO [Member]
Related party transactions [Line Items]
Non-controlling interests		2,700			2,700			5,300
Participation in multi-borrower financing facility [Member]
Related party transactions [Line Items]
Outstanding debt of related party		146,700			146,700			106,600
Due from Related Parties		12			12			23,400
Participation in Multi-Seller Consumer Loan Securitization [Member] | Repurchase Agreements [Member] | Acquiror Repurchase Agreement [Member]
Related party transactions [Line Items]
Fair value repurchase agreements		5,700			5,700			5,700
Investments held with dealers as collateral	[7],[9],[10]	9,400			9,400			9,400
Loan receivable from affiliated entity related to warehouse facility [Member]
Related party transactions [Line Items]
Other assets		21,000			21,000			$ 16,900
Mortgage-related Commercial [Member] | Corporate Equity Investments [Member]
Related party transactions [Line Items]
Investments Fair Value		$ 1,200			$ 1,200
Ownership percentage of limited liability company held as investment		17.00%			17.00%

[1] See Note 9 for further details on management fee rebates.

[2] In the Company's Consolidated Statement of Assets, Liabilities, and Equity, all balances associated with repurchase agreements, reverse repurchase agreements, and financial derivatives are presented on a gross basis.

[3] Includes non-performing commercial mortgage loans in the amount of $23.9 million whereby principal and/or interest is past due and a maturity date is not applicable.

[4] Loans and real estate owned are beneficially owned by the Company through participation certificates in the various trusts that hold such investments. See Note 9 to the Notes to Consolidated Financial Statements.

[5] See Note 2 and Note 3 in Notes to Consolidated Financial Statements.

[6] The table below shows the ratings on the Company's long investments from Moody's, Standard and Poor's, or Fitch, as well as the Company's long investments that were unrated but guaranteed by the Federal National Mortgage Association, the Federal Home Loan Mortgage Corporation, or the Government National Mortgage Association. Ratings tend to be a lagging credit indicator; as a result, the credit quality of the Company's long investment holdings may be lower than the credit quality implied based on the ratings listed below. In situations where an investment has a split rating, the lowest provided rating is used. The ratings descriptions include ratings qualified with a "+," "-," "1," "2," or "3."Rating Description Percent of EquityUnrated but Agency-Guaranteed 160.32%A/A/A 0.81%Baa/BBB/BBB 2.62%Ba/BB/BB or below 68.03%Unrated 101.85%

[7] See Note 2 and Note 3 in Notes to Consolidated Financial Statements.

[8] The table below shows the Company's long investment ratings from Moody's, Standard and Poor's, or Fitch, as well as the Company's long investments that were unrated but guaranteed by the Federal National Mortgage Association, the Federal Home Loan Mortgage Corporation, or the Government National Mortgage Association. Ratings tend to be a lagging credit indicator; as a result, the credit quality of the Company's long investment holdings may be lower than the credit quality implied based on the ratings listed below. In situations where an investment has a split rating, the lowest provided rating is used. The ratings descriptions include ratings qualified with a "+," "-," "1," "2," or "3."Rating Description Percent of EquityUnrated but Agency-Guaranteed 206.36%Aaa/AAA/AAA 11.52%Aa/AA/AA 0.85%A/A/A 1.04%Baa/BBB/BBB 4.71%Ba/BB/BB or below 67.62%Unrated 136.01%

[9] Represents the Company's investment in a related party. See Note 9 to the Notes to Consolidated Financial Statements

[10] Includes $107.9 million of non-qualified mortgage loans that have been securitized and are held in a consolidated securitization trust. See Note 6 to the Notes to Consolidated Financial Statements.