Other Assets (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Jun. 30, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Dec. 31, 2018
Other Assets [Abstract]
REO	$ 298,866	$ 348,516	$ 411,659	$ 376,200	$ 334,069	$ 249,413
Capital contributions made to loan origination partners	108,887		147,992
Other interest-earning assets	45,442		70,468
Interest receivable	42,723		70,986
Other MBS and loan related receivables	36,342		43,842
Other	39,354		39,304
Total Other Assets	571,614		784,251
Real estate held-for-investment	$ 59,300		$ 27,300