Other Liabilities (Details) - USD ($) $ in Thousands	Mar. 31, 2021	Dec. 31, 2020
Other Liabilities [Abstract]
Payable for unsettled residential whole loans purchases	$ 112,202	$ 0
Dividends and dividend equivalents payable	33,640	34,016
Accrued interest payable	10,948	11,116
Accrued expenses and other	22,922	25,390
Total Other Liabilities	$ 179,712	$ 70,522