Fair Value of Financial Instruments - Portion of Net Gains (Losses) Attributable to Level 3 Assets and Liabilities Still Held Included in Net Income (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015
Loan purchase commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 liabilities still held included in net income	$ 0	$ 9,736	$ 0	$ 9,806
Commercial Secured Borrowings
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 liabilities still held included in net income	0	(454)	0	750
Asset-backed securities issued
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 liabilities still held included in net income	10,522	300	(14,419)	(6,198)
Residential Loans Held For Investment at Fair Value | Residential loans at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	3,818	16,451	32,202	12,115
Residential Loans Held For Investment at Fair Value | Residential loans at consolidated Sequoia entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	9,200	(419)	(18,864)	4,912
Commercial Loans at Fair Value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	0	3,175	0	1,971
Trading securities retained, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	8,646	(8,298)	978	(13,274)
AFS securities retained, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	0	(226)	(305)	(226)
MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	6,549	(25,523)	(36,738)	(15,989)
Loan purchase commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	5,381	0	5,896	0
Guarantee Asset
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	$ 307	$ (1,098)	$ (2,070)	$ (1,799)