Debt, net Table of Debt (Details) - USD ($) $ in Thousands		3 Months Ended
		Mar. 31, 2018	Dec. 31, 2017
Debt Instrument [Line Items]
Long-term Debt, Gross		$ 330,212	$ 356,104
Debt Instrument, Unamortized Discount		(211)	(191)
Deferred Financing Costs, Net		(9,493)	(9,832)
Debt, net		320,508	346,081
Secured Debt
Debt Instrument [Line Items]
Long-term Debt, Gross		28,000	28,500
Long-term Debt, Other Disclosures [Abstract]
Debt instrument, maximum borrowing capacity		$ 155,000
Secured Debt | London Interbank Offered Rate (LIBOR)
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Description of Variable Rate Basis		LIBOR
Junior subordinated notes
Debt Instrument [Line Items]
Long-term Debt, Gross		$ 125,000	125,000
Long-term Debt, Other Disclosures [Abstract]
Debt instrument, maximum borrowing capacity		125,000
Trust Preferred Securities Subject to Mandatory Redemption [Member]
Debt Instrument [Line Items]
Long-term Debt, Gross		35,000	35,000
Long-term Debt, Other Disclosures [Abstract]
Debt instrument, maximum borrowing capacity		$ 35,000
Trust Preferred Securities Subject to Mandatory Redemption [Member] | London Interbank Offered Rate (LIBOR)
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Description of Variable Rate Basis		LIBOR
Corporate Debt Payable
Debt Instrument [Line Items]
Long-term Debt, Gross		$ 188,000	188,500
Asset based revolving financing (1) (2)
Debt Instrument [Line Items]
Long-term Debt, Gross	[1],[2]	103,417	$ 118,794
Long-term Debt, Other Disclosures [Abstract]
Debt instrument, maximum borrowing capacity		$ 175,000
Long-term Debt, Weighted Average Interest Rate		4.16%	3.73%
Asset based revolving financing (1) (2) | London Interbank Offered Rate (LIBOR)
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Description of Variable Rate Basis		LIBOR
Residential mortgage warehouse borrowings (3)
Debt Instrument [Line Items]
Long-term Debt, Gross	[3]	$ 38,795	$ 48,810
Long-term Debt, Other Disclosures [Abstract]
Debt instrument, maximum borrowing capacity	[3]	$ 76,000
Long-term Debt, Weighted Average Interest Rate		4.55%	3.70%
Residential mortgage warehouse borrowings (3) | London Interbank Offered Rate (LIBOR)
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Description of Variable Rate Basis		LIBOR
Asset Backed Debt Payable
Debt Instrument [Line Items]
Long-term Debt, Gross		$ 142,212	$ 167,604
Minimum | Secured Debt | London Interbank Offered Rate (LIBOR)
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Basis Spread on Variable Rate		1.00%
Minimum | Asset based revolving financing (1) (2) | London Interbank Offered Rate (LIBOR)
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Basis Spread on Variable Rate	[1],[2]	2.25%
Minimum | Residential mortgage warehouse borrowings (3) | London Interbank Offered Rate (LIBOR)
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Basis Spread on Variable Rate	[3]	2.50%
Maximum | Secured Debt | London Interbank Offered Rate (LIBOR)
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Basis Spread on Variable Rate		6.50%
Maximum | Junior subordinated notes
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Interest Rate, Stated Percentage		8.50%
Maximum | Trust Preferred Securities Subject to Mandatory Redemption [Member]
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Interest Rate, Stated Percentage		4.10%
Maximum | Asset based revolving financing (1) (2) | London Interbank Offered Rate (LIBOR)
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Basis Spread on Variable Rate	[1],[2]	2.60%
Maximum | Residential mortgage warehouse borrowings (3) | London Interbank Offered Rate (LIBOR)
Long-term Debt, Other Disclosures [Abstract]
Debt Instrument, Basis Spread on Variable Rate	[3]	3.25%

[1] Asset based revolving financing is generally recourse only to specific assets and related cash flows.

[2] The weighted average coupon rate for asset based revolving financing was 4.16% and 3.73% at March 31, 2018 and December 31, 2017, respectively.

[3] The weighted average coupon rate for residential mortgage warehouse borrowings was 4.55% and 3.70% at March 31, 2018 and December 31, 2017, respectively.