Residential Mortgage Securities and MSR-Related Assets (MSR Related Assets - Narrative) (Details) - USD ($)	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019	Dec. 31, 2019	Dec. 31, 2018
Term notes backed by MSR-related collateral
Debt Securities, Available-for-sale [Line Items]
Term notes backed by MSR related collateral	$ 706,608,000		$ 1,157,463,000
Amortized costs	$ 706,600,000		1,200,000,000
Weighted average yield	4.74%	4.75%
Weighted average to maturity	5 years 1 month 6 days	5 years 3 months 18 days
Proceeds from sale of MSR	$ 136,800,000
Realized losses	(24,600,000)
Impairment	280,800,000
Gross unrealized gains			$ 5,200,000
Corporate Loan
Debt Securities, Available-for-sale [Line Items]
Commitment to lend				$ 100,000,000.0
Amount drawn	$ 33,800,000
Interest coupon rate	3.93%
Debt term	5 months
Commitment period on undrawn amount	5 months
Minimum | Corporate Loan
Debt Securities, Available-for-sale [Line Items]
Commitment fee on undrawn amount	0.25%
Maximum | Corporate Loan
Debt Securities, Available-for-sale [Line Items]
Commitment fee on undrawn amount	1.00%