Here is another update. I managed to build xserver-xorg with a minimum amount of patches (basicly just debian/patches/125-127) and after installing that and installing the 100.14.11 nvidia driver it does no longer crash. So it seems like on of our patches breaks nvidia.
Here is another update. I managed to build xserver-xorg with a minimum amount of patches (basicly just debian/ patches/ 125-127) and after installing that and installing the 100.14.11 nvidia driver it does no longer crash. So it seems like on of our patches breaks nvidia.