It appears the problem is simply that /usr/lib/xorg/modules/extensions/ is no longer there when xserver-xorg-core is removed and dpkg fails over if the dir is not there.
It appears the problem is simply that /usr/lib/ xorg/modules/ extensions/ is no longer there when xserver-xorg-core is removed and dpkg fails over if the dir is not there.