Cannot switch users after changing display driver

[dee veloper]

Steps to Reproduce:

Have more than one user.
Be logged into one of the users after a restart.
Open additional drivers and change your Nvidia display driver to a different one (i changed to the open source driver).
Apply the change.
Attempt to "Switch User"
When you try to log in with the second user after entering the password, it just takes you back to select a user and you can't log in to the other user.
Log back in to the initial user and switch the driver back to the one it was on before you switched.
Attempt to "Switch User"
It works

Ubuntu 20.04.3 LTS
Intel Core i5-11600K
64GB RAM
Nvidia GTX 1070 (currently on Nvidia 510 Driver)

I discovered this while I was trying to solve a different issue of my computer not waking from a simple blank screen.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Libera.chat.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1961059/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]