Other Assets (Details) - USD ($) $ in Thousands	Mar. 31, 2021	Dec. 31, 2020	Mar. 31, 2020	Dec. 31, 2019
Other Assets [Abstract]
REO	$ 220,393	$ 249,699	$ 411,473	$ 411,659
Capital contributions made to loan origination partners	81,374	47,148
Interest receivable	35,989	38,850
Other MBS and loan related receivables	23,403	16,682
Other	31,567	33,002
Total Other Assets	392,726	385,381
Real estate held-for-investment	$ 61,100	$ 61,800