Borrowings - Assets Sold Under Agreement to Repurchase (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2018	Dec. 31, 2017	Dec. 31, 2016
Carrying value:
Unpaid principal balance	$ 1,935,200
Unamortized debt issuance costs and premiums	(1,341)	$ 672
Total loans sold under agreements to repurchase	1,933,859	2,381,538
Fair value of assets pledged to secure	1,923,857	2,530,299
Servicing advances	162,895	114,643
Assets sold under agreements to repurchase
During the period:
Average balance of mortgage loans sold under agreements to repurchase	$ 1,626,729	$ 1,829,257	$ 1,438,181
Weighted-average interest rate (as a percent)	3.87%	3.18%	2.91%
Total interest expense	$ 22,463	$ 60,286	$ 49,791
Maximum daily amount outstanding	2,380,121	3,022,656	2,661,746
Carrying value:
Unpaid principal balance	1,935,200	2,380,866
Unamortized debt issuance costs and premiums	(1,341)	672
Total loans sold under agreements to repurchase	$ 1,933,859	$ 2,381,538
Weighted average interest rate (as a percent)	4.22%	3.24%
Available borrowing capacity committed	$ 695,767	$ 316,503
Available borrowing capacity uncommitted	2,354,033	2,257,631
Available borrowing capacity	3,049,800	2,574,134
Margin deposits placed with counterparties	3,750	3,750
Amortization of debt issuance costs	40,500	1,300	$ 7,300
Amount of Master Repurchase Agreement incentives to refinance include in interest expense	$ 48,100	9,200
Initial term of Master Repurchase Agreement	6 months
Assets sold under agreements to repurchase | Mortgage Loans held for sale
Carrying value:
Fair value of assets pledged to secure	$ 1,923,857	2,530,299
Assets sold under agreements to repurchase | Servicing advances
Carrying value:
Servicing advances	162,895	114,643
Assets sold under agreements to repurchase | Financing receivable | PMT
Carrying value:
Fair value of assets pledged to secure	131,025	144,128
Assets sold under agreements to repurchase | Mortgage servicing rights
Carrying value:
Fair value of assets pledged to secure	$ 2,807,333	$ 2,098,067