Other Assets (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Mar. 31, 2020	Dec. 31, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018
Other Assets [Abstract]
REO	$ 348,516	$ 411,473	$ 411,659	$ 334,069	$ 290,587	$ 249,413
Capital contributions made to loan origination partners	109,585		147,992
Other interest-earning assets	49,971		70,468
Interest receivable	49,588		70,986
Other MBS and loan related receivables	24,471		43,842
Other	31,343		39,304
Total Other Assets	613,474		784,251
Real estate held-for-investment	$ 53,000		$ 27,300