Debt, net - Narrative (Details) $ in Thousands			12 Months Ended
		May 04, 2018 USD ($) Rate	Dec. 31, 2018 USD ($) Rate	Dec. 31, 2017 USD ($)	Dec. 31, 2016 USD ($)	Nov. 28, 2018 USD ($)	Apr. 30, 2017 USD ($)
Debt Instrument [Line Items]
Long-term Debt, Gross			$ 364,273	$ 356,104
Repayments of Secured Debt			1,617,346	1,816,537	$ 1,957,183
Asset-based revolving facility
Debt Instrument [Line Items]
Debt instrument, maximum borrowing capacity			130,000
Long-term Debt, Gross	[1],[2]		86,092	118,794
Residential mortgage warehouse borrowings
Debt Instrument [Line Items]
Debt instrument, maximum borrowing capacity			76,000
Long-term Debt, Gross	[1],[3]		46,091	48,810
Junior subordinated notes
Debt Instrument [Line Items]
Debt instrument, maximum borrowing capacity			125,000
Long-term Debt, Gross			125,000	125,000
Secured corporate credit agreements
Debt Instrument [Line Items]
Increase (Decrease) in maximum borrowing capacity		$ 47,000
Debt instrument, maximum borrowing capacity		$ 75,000	132,090
Long-term Debt, Gross			72,090	28,500
Preferred trust securities
Debt Instrument [Line Items]
Debt instrument, maximum borrowing capacity			35,000
Long-term Debt, Gross			35,000	35,000
Corporate and Other | Secured corporate credit agreements
Debt Instrument [Line Items]
Long-term Debt, Gross			72,090	28,500
Specialty Insurance | Asset-based revolving facility
Debt Instrument [Line Items]
Long-term Debt, Gross			0	0
Specialty Insurance | Junior subordinated notes
Debt Instrument [Line Items]
Long-term Debt, Gross			$ 125,000
Debt Instrument, Potential Deferred Interest Periods			5 years
Specialty Insurance | Secured corporate credit agreements
Debt Instrument [Line Items]
Debt instrument, maximum borrowing capacity			$ 30,000
Debt instrument, accordion feature			30,000
Specialty Insurance | Preferred trust securities
Debt Instrument [Line Items]
Long-term Debt, Gross			35,000
Asset Based Leverage - Corporate Loans | Specialty Insurance | Asset-based revolving facility
Debt Instrument [Line Items]
Debt instrument, maximum borrowing capacity			105,000			$ 150,000
Long-term Debt, Gross			81,343	101,428
Asset Based Leverage - Nonperforming Loans | Specialty Insurance | Asset-based revolving facility
Debt Instrument [Line Items]
Repayments of Secured Debt			11,917
Asset Backed Revolving Line of Credit - Premium Financing | Specialty Insurance | Asset-based revolving facility
Debt Instrument [Line Items]
Debt instrument, maximum borrowing capacity			25,000				$ 15,000
Long-term Debt, Gross			4,749	5,449
Warehouse Agreement Borrowings | Residential mortgage warehouse borrowings
Debt Instrument [Line Items]
Long-term Debt, Gross	[1],[3]		$ 46,091	48,810
Number of credit agreements			3
Line of Credit Facility, Maximum Borrowing Capacity	[1],[3]		$ 76,000
Maximum | Junior subordinated notes
Debt Instrument [Line Items]
Debt Instrument, Interest Rate, Stated Percentage | Rate			8.50%
Maximum | Preferred trust securities
Debt Instrument [Line Items]
Debt Instrument, Interest Rate, Stated Percentage | Rate			4.10%
Maximum | Specialty Insurance | Junior subordinated notes
Debt Instrument [Line Items]
Debt Instrument, Interest Rate, Stated Percentage | Rate			8.50%
Maximum | Specialty Insurance | Preferred trust securities
Debt Instrument [Line Items]
Debt Instrument, Interest Rate, Stated Percentage | Rate			4.10%
London Interbank Offered Rate (LIBOR) | Minimum | Asset-based revolving facility
Debt Instrument [Line Items]
Debt Instrument, Basis Spread on Variable Rate | Rate	[1],[2]		2.00%
London Interbank Offered Rate (LIBOR) | Minimum | Residential mortgage warehouse borrowings
Debt Instrument [Line Items]
Debt Instrument, Basis Spread on Variable Rate | Rate	[3]		2.00%
London Interbank Offered Rate (LIBOR) | Minimum | Secured corporate credit agreements
Debt Instrument [Line Items]
Debt Instrument, Basis Spread on Variable Rate | Rate		1.25%	1.00%
London Interbank Offered Rate (LIBOR) | Maximum | Asset-based revolving facility
Debt Instrument [Line Items]
Debt Instrument, Basis Spread on Variable Rate | Rate	[1],[2]		2.60%
London Interbank Offered Rate (LIBOR) | Maximum | Residential mortgage warehouse borrowings
Debt Instrument [Line Items]
Debt Instrument, Basis Spread on Variable Rate | Rate	[3]		3.00%
London Interbank Offered Rate (LIBOR) | Maximum | Secured corporate credit agreements
Debt Instrument [Line Items]
Debt Instrument, Basis Spread on Variable Rate | Rate		5.50%	5.50%
London Interbank Offered Rate (LIBOR) | Maximum | Specialty Insurance | Secured corporate credit agreements
Debt Instrument [Line Items]
Debt Instrument, Basis Spread on Variable Rate | Rate			1.00%
London Interbank Offered Rate (LIBOR) | Maximum | Asset Based Leverage - Corporate Loans | Specialty Insurance | Asset-based revolving facility
Debt Instrument [Line Items]
Debt Instrument, Basis Spread on Variable Rate | Rate			2.00%
London Interbank Offered Rate (LIBOR) | Maximum | Asset Backed Revolving Line of Credit - Premium Financing | Specialty Insurance | Asset-based revolving facility
Debt Instrument [Line Items]
Debt Instrument, Basis Spread on Variable Rate | Rate			2.60%
Luxury Disposition | Warehouse Agreement Borrowings | Residential mortgage warehouse borrowings
Debt Instrument [Line Items]
Long-term Debt, Gross			$ 61,381	$ 53,835
Number of credit agreements			3
Line of Credit Facility, Maximum Borrowing Capacity	[1],[3]		$ 95,000

[1] Asset based debt is generally recourse only to specific assets and related cash flows.

[2] The weighted average coupon rate for asset based revolving financing was 4.30% and 4.17% at December 31, 2018 and December 31, 2017, respectively.

[3] The weighted average coupon rate for residential mortgage warehouse borrowings was 4.66% and 4.27% at December 31, 2018 and December 31, 2017, respectively.