Related Party Transactions (Details) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016 USD ($)	Sep. 30, 2015 USD ($)	Sep. 30, 2016 USD ($)	Sep. 30, 2015 USD ($)	Dec. 31, 2015 USD ($)
Financial Instruments Owned and Pledged as Collateral [Line Items]
Securitized debt	$ 30,771		$ 30,771		$ 0
Annual base management fee percentage			1.50%
Management fee description			The Operating Partnership pays the Manager 1.50% per annum of total equity of the Operating Partnership calculated in accordance with U.S. GAAP as of the end of each fiscal quarter (before deductions for base management fees and incentive fees payable with respect to such fiscal quarter), provided that total equity is adjusted to exclude one-time events pursuant to changes in U.S. GAAP, as well as non-cash charges after discussion between the Manager and the Company's independent directors, and approval by a majority of the Company's independent directors in the case of non-cash charges.
Management Fee Expense	2,485	$ 2,849	$ 7,649	$ 8,720
Incentive fee rate			25.00%
Loss Carryforward	$ 17,500		$ 17,500
Incentive fees arrangement			The Manager is entitled to receive a quarterly incentive fee equal to the positive excess, if any, of (i) the product of (A) 25% and (B) the excess of (1) Adjusted Net Income (described below) for the Incentive Calculation Period (which means such fiscal quarter and the immediately preceding three fiscal quarters) over (2) the sum of the Hurdle Amounts (described below) for the Incentive Calculation Period, over (ii) the sum of the incentive fees already paid or payable for each fiscal quarter in the Incentive Calculation Period preceding such fiscal quarter. For purposes of calculating the incentive fee, "Adjusted Net Income" for the Incentive Calculation Period means the net increase in equity from operations of the Operating Partnership, after all base management fees but before any incentive fees for such period, and excluding any non-cash equity compensation expenses for such period, as reduced by any Loss Carryforward (as described below) as of the end of the fiscal quarter preceding the Incentive Calculation Period. For purposes of calculating the incentive fee, the "Loss Carryforward" as of the end of any fiscal quarter is calculated by determining the excess, if any, of (1) the Loss Carryforward as of the end of the immediately preceding fiscal quarter over (2) the Company's net increase in equity from operations (expressed as a positive number) or net decrease in equity from operations (expressed as a negative number) of the Operating Partnership for such fiscal quarter. As of September 30, 2016 there was a Loss Carryforward of $17.5 million. For purposes of calculating the incentive fee, the "Hurdle Amount" means, with respect to any fiscal quarter, the product of (i) one-fourth of the greater of (A) 9% and (B) 3% plus the 10-year U.S. Treasury rate for such fiscal quarter, (ii) the sum of (A) the weighted average gross proceeds per share of all common share and OP Unit issuances since inception of the Company and up to the end of such fiscal quarter, with each issuance weighted by both the number of shares and OP Units issued in such issuance and the number of days that such issued shares and OP Units were outstanding during such fiscal quarter, using a first-in first-out basis of accounting (i.e. attributing any share and OP Unit repurchases to the earliest issuances first) and (B) the result obtained by dividing (I) retained earnings attributable to common shares and OP Units at the beginning of such fiscal quarter by (II) the average number of common shares and OP Units outstanding for each day during such fiscal quarter, (iii) the sum of the average number of common shares, LTIP Units, and OP Units outstanding for each day during such fiscal quarter. For purposes of determining the Hurdle Amount, issuances of common shares and OP Units (a) as equity incentive awards, (b) to the Manager as part of its base management fee or incentive fee and (c) to the Manager or any of its affiliates in privately negotiated transactions, are excluded from the calculation. The payment of the incentive fee will be in a combination of common shares and cash, provided that at least 10% of any quarterly payment will be made in common shares.
Incentive fee hurdle rate fixed			9.00%
Incentive fee hurdle rate floating			3.00%
Minimum percentage of incentive fee to be paid in share			10.00%
Expense Reimbursement Period			60 days
Expense Reimbursement - Manager			$ 5,200	$ 2,900
Number of investments in mortgage originators that represent related party transactions	2		2
Warehouse facility, monetary amount	$ 5,000		$ 5,000
Warehouse facility, interest rate			5.00%
Advances made under warehouse facility	600		$ 600
Outstanding debt of related party trust	70,300		$ 70,300
Termination Fee, Description			The Management Agreement requires the Company to pay a termination fee to the Manager in the event of (1) the Company's termination or non-renewal of the Management Agreement without cause or (2) the Company's termination of the Management Agreement based on unsatisfactory performance by the Manager that is materially detrimental to the Company (3) the Manager's termination of the Management Agreement upon a default by the Company in the performance of any material term of the Management Agreement. Such termination fee will be equal to the amount of three times the sum of (i) the average annual Quarterly Base Management Fee Amounts paid or payable with respect to the two 12-month periods ending on the last day of the latest fiscal quarter completed on or prior to the date of the notice of termination or non-renewal and (ii) the average annual Quarterly Incentive Fee Amounts paid or payable with respect to the two 12-month periods ending on the last day of the latest fiscal quarter completed on or prior to the date of the notice of termination or non-renewal.
Aggregate unpaid principal balance of loans sold			$ 124,000
Company's share of loans sold			51.00%
Co-participants share of loans sold			49.00%
Senior notes issued in securitization transaction			$ 87,000
Subordinated notes issued in securitization transaction			$ 18,700
Company's percentage ownership of jointly owned entity			51.00%
Affiliated co-participants percentage ownership of jointly owned entity			49.00%
Related Party 1 [Member]
Financial Instruments Owned and Pledged as Collateral [Line Items]
Fair value of loans held in related party trust	6,200		$ 6,200		$ 3,800
Related Party 2 [Member]
Financial Instruments Owned and Pledged as Collateral [Line Items]
Fair value of loans held in related party trust	$ 67,900		$ 67,900