Related Party Transactions (Details) - USD ($) $ in Thousands		3 Months Ended
		Mar. 31, 2017		Mar. 31, 2016	Dec. 31, 2016
Financial Instruments Owned and Pledged as Collateral [Line Items]
Other secured borrowings		$ 61,802			$ 24,086
Annual base management fee percentage		1.50%
Management fee description		The Operating Partnership pays the Manager 1.50% per annum of total equity of the Operating Partnership calculated in accordance with U.S. GAAP as of the end of each fiscal quarter (before deductions for base management fees and incentive fees payable with respect to such fiscal quarter), provided that total equity is adjusted to exclude one-time events pursuant to changes in U.S. GAAP, as well as non-cash charges after discussion between the Manager and the Company's independent directors, and approval by a majority of the Company's independent directors in the case of non-cash charges.
Management Fee Expense		$ 2,410		$ 2,611
Incentive fee rate		25.00%
Loss Carryforward		$ 200
Incentive fees arrangement		The Manager is entitled to receive a quarterly incentive fee equal to the positive excess, if any, of (i) the product of (A) 25% and (B) the excess of (1) Adjusted Net Income (described below) for the Incentive Calculation Period (which means such fiscal quarter and the immediately preceding three fiscal quarters) over (2) the sum of the Hurdle Amounts (described below) for the Incentive Calculation Period, over (ii) the sum of the incentive fees already paid or payable for each fiscal quarter in the Incentive Calculation Period preceding such fiscal quarter. For purposes of calculating the incentive fee, "Adjusted Net Income" for the Incentive Calculation Period means the net increase in equity from operations of the Operating Partnership, after all base management fees but before any incentive fees for such period, and excluding any non-cash equity compensation expenses for such period, as reduced by any Loss Carryforward (as described below) as of the end of the fiscal quarter preceding the Incentive Calculation Period. For purposes of calculating the incentive fee, the "Loss Carryforward" as of the end of any fiscal quarter is calculated by determining the excess, if any, of (1) the Loss Carryforward as of the end of the immediately preceding fiscal quarter over (2) the Company's net increase in equity from operations (expressed as a positive number) or net decrease in equity from operations (expressed as a negative number) of the Operating Partnership for such fiscal quarter. As of March 31, 2017, there was a Loss Carryforward of $0.2 million. For purposes of calculating the incentive fee, the "Hurdle Amount" means, with respect to any fiscal quarter, the product of (i) one-fourth of the greater of (A) 9% and (B) 3% plus the 10-year U.S. Treasury rate for such fiscal quarter, (ii) the sum of (A) the weighted average gross proceeds per share of all common share and OP Unit issuances since inception of the Company and up to the end of such fiscal quarter, with each issuance weighted by both the number of shares and OP Units issued in such issuance and the number of days that such issued shares and OP Units were outstanding during such fiscal quarter, using a first-in first-out basis of accounting (i.e. attributing any share and OP Unit repurchases to the earliest issuances first) and (B) the result obtained by dividing (I) retained earnings attributable to common shares and OP Units at the beginning of such fiscal quarter by (II) the average number of common shares and OP Units outstanding for each day during such fiscal quarter, (iii) the sum of the average number of common shares, LTIP Units, and OP Units outstanding for each day during such fiscal quarter. For purposes of determining the Hurdle Amount, issuances of common shares and OP Units (a) as equity incentive awards, (b) to the Manager as part of its base management fee or incentive fee and (c) to the Manager or any of its affiliates in privately negotiated transactions, are excluded from the calculation. The payment of the incentive fee will be in a combination of common shares and cash, provided that at least 10% of any quarterly payment will be made in common shares.
Incentive fee hurdle rate fixed		9.00%
Incentive fee hurdle rate floating		3.00%
Minimum percentage of incentive fee to be paid in share		10.00%
Expense Reimbursement Period		60 days
Expense Reimbursement - Manager		$ 1,300		$ 2,000
Termination Fee, Description		The Management Agreement requires the Company to pay a termination fee to the Manager in the event of (1) the Company's termination or non-renewal of the Management Agreement without cause or (2) the Company's termination of the Management Agreement based on unsatisfactory performance by the Manager that is materially detrimental to the Company (3) the Manager's termination of the Management Agreement upon a default by the Company in the performance of any material term of the Management Agreement. Such termination fee will be equal to the amount of three times the sum of (i) the average annual Quarterly Base Management Fee Amounts paid or payable with respect to the two 12-month periods ending on the last day of the latest fiscal quarter completed on or prior to the date of the notice of termination or non-renewal and (ii) the average annual Quarterly Incentive Fee Amounts paid or payable with respect to the two 12-month periods ending on the last day of the latest fiscal quarter completed on or prior to the date of the notice of termination or non-renewal.
Warehouse facility, monetary amount		$ 5,000
Line of Credit Facility, Interest Rate at Period End		10.00%
Advances made under warehouse facility		$ 2,100
Purchase commitment principal balance		22,200
Fair Value		1,864,213			1,505,026
Estimated remaining contingent purchase obligations		241,900
Non-controlling interests		18,204			7,116
Outstanding debt of related party trust		53,800
Aggregate unpaid principal balance of loans sold		$ 124,000
Company's share of loans sold		51.00%
Co-participants share of loans sold		49.00%
Senior notes issued in securitization transaction		$ 87,000
Subordinated notes issued in securitization transaction		$ 18,700
Company's percentage ownership of jointly owned entity		51.00%
Affiliated co-participants percentage ownership of jointly owned entity		49.00%
Fair value repurchase agreements		$ 293,802			184,819
Investments held with dealers as collateral		1,220,000			1,150,000
Repurchase Agreements [Member]
Financial Instruments Owned and Pledged as Collateral [Line Items]
Fair value repurchase agreements		293,802	[1],[2]		184,819	[3],[4]
Repurchase Agreements [Member] | Acquiror Repurchase Agreement [Member]
Financial Instruments Owned and Pledged as Collateral [Line Items]
Fair value repurchase agreements		6,166
Investments held with dealers as collateral	[3],[4],[5]	9,400
Residential mortgage loans and REO [Member]
Financial Instruments Owned and Pledged as Collateral [Line Items]
Fair value of loans held in related party trust		113,900			86,000
Commercial mortgage loans and REO [Member]
Financial Instruments Owned and Pledged as Collateral [Line Items]
Other secured borrowings		18,200			24,100
Fair Value	[2],[6],[7],[8]	23,400
Consumer loans and asset-backed securities backed by consumer loans [Member]
Financial Instruments Owned and Pledged as Collateral [Line Items]
Fair value of loans held in related party trust		86,700			43,200
Fair Value		114,278	[2],[7],[8]		114,472	[4],[9],[10]
Related Party 1 [Member]
Financial Instruments Owned and Pledged as Collateral [Line Items]
Fair value of loans held in related party trust		7,400			$ 7,600
Unrelated third party joint venture interest [Member] | Non-controlling interest of joint venture partner [Member] | Commercial mortgage loans and REO [Member]
Financial Instruments Owned and Pledged as Collateral [Line Items]
Non-controlling interests		2,400
Related party joint venture interest [Member] | Non-controlling interest of joint venture partner [Member] | Commercial mortgage loans and REO [Member]
Financial Instruments Owned and Pledged as Collateral [Line Items]
Non-controlling interests		$ 8,600

[1] In general, securities received pursuant to repurchase agreements were delivered to counterparties in short sale transactions.

[2] See Note 2 and Note 3 in Notes to Consolidated Financial Statements.

[3] In general, securities received pursuant to repurchase agreements were delivered to counterparties in short sale transactions.

[4] See Note 2 and Note 3 in Notes to Consolidated Financial Statements.

[5] Repurchase agreement is between the Company and CILO 2016-LD1 Holdings LLC, an entity in which the Company has a beneficial interest and is co-owned by an affiliate of Ellington Management Group, L.L.C. CILO 2016-LD1 Holdings LLC owns subordinated notes issued by, as well as trust certificates representing ownership of, a securitization trust. See Note 7 to the Notes to Consolidated Financial Statements.

[6] Includes non-performing commercial loans in the amount of $6.5 million whereby principal and/or interest is past due and a maturity date is not applicable.

[7] Loans and real estate owned are beneficially owned by the Company through participation certificates in the various trusts that hold such investments. See Note 7 to the Notes to the Consolidated Financial Statements.

[8] The table below shows the ratings on the Company's long investments from Moody's, Standard and Poor's, or Fitch, as well as the Company's long investments that were unrated but guaranteed by the Federal National Mortgage Association, the Federal Home Loan Mortgage Corporation, or the Government National Mortgage Association. Ratings tend to be a lagging credit indicator; as a result, the credit quality of the Company's long investment holdings may be lower than the credit quality implied based on the ratings listed below. In situations where an investment has a split rating, the lowest provided rating is used. The ratings descriptions include ratings qualified with a "+," "-," "1," "2," or "3."Rating Description Percent of EquityUnrated but Agency-Guaranteed 167.23%A/A/A 0.05%Aaa/AAA/AAA 5.57%Baa/BBB/BBB 3.76%Ba/BB/BB or below 39.76%Unrated 68.48%

[9] Loans and real estate owned are beneficially owned by the Company through participation certificates in the various trusts that hold such investments. See Note 7 to the Notes to Consolidated Financial Statements.

[10] The table below shows the Company's long investment ratings from Moody's, Standard and Poor's, or Fitch, as well as the Company's long investments that were unrated but guaranteed by the Federal National Mortgage Association, the Federal Home Loan Mortgage Corporation, or the Government National Mortgage Association. Ratings tend to be a lagging credit indicator; as a result, the credit quality of the Company's long investment holdings may be lower than the credit quality implied based on the ratings listed below. In situations where an investment has a split rating, the lowest provided rating is used. The ratings descriptions include ratings qualified with a "+," "-," "1," "2," or "3."Rating Description Percent of EquityUnrated but Agency-Guaranteed 139.27%Aaa/AAA/AAA 0.84%Aa/AA/AA 0.03%A/A/A 0.05%Baa/BBB/BBB 2.60%Ba/BB/BB or below 30.24%Unrated 60.39%