Comment 2 for bug 380360

Another reasonable workaround for your case would be to not run ssh-agent, for instance by disabling it in /etc/X11/Xsession.options

For a proper fix, please report a bug in Debian. I don't think Ubuntu wants to divert from Debian in order to accommodate this unusual usage pattern.