Installing i386 and amd64 PAM stacks causes shutdown/logout/etc. to break

[Julien Aubin]

Hi,

After I upgraded to Ubuntu Trusty, I've seen that it is no longer possible to suspend/shutdown/restart my system from the GUI. In other words within lightdm buttons to do so are grayed out, and within KDE it is the same. I had the same bug in Saucy beta and the culprit was the login component, and after its upgrade the issue was fixed.

My system :
Kubuntu 14.04 LTS AMD64
AMD Phenom 9850
8 GB RAM
NVidia GeForce 660 Ti w/ NVidia 331 driver

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: login 1:4.1.5.1-1ubuntu9
ProcVersionSignature: Ubuntu 3.13.0-18.38-generic 3.13.6
Uname: Linux 3.13.0-18-generic x86_64
NonfreeKernelModules: nvidia
ApportVersion: 2.13.3-0ubuntu1
Architecture: amd64
CurrentDesktop: KDE
Date: Fri Mar 21 07:21:52 2014
ProcEnviron:
 LANGUAGE=fr_FR
 TERM=xterm
 PATH=(custom, no user)
 LANG=fr_FR.UTF-8
 SHELL=/bin/bash
SourcePackage: shadow
UpgradeStatus: Upgraded to trusty on 2014-03-20 (0 days ago)

[Martin Pitt (pitti)]

As your normal user in a GUI session, can you please give me the output of

  gdbus call -y -d org.freedesktop.login1 -o /org/freedesktop/login1 -m org.freedesktop.login1.Manager.CanReboot

That ought to be something like "('yes',)".

If that says yes, can you confirm that rebooting works with

  gdbus call -y -d org.freedesktop.login1 -o /org/freedesktop/login1 -m org.freedesktop.login1.Manager.Reboot true

Thanks!

[Martin Pitt (pitti)]

Oh, if it doesn't work, please copy&paste the entire output, there should be an error message (like permission denied by polkit or something). Please also give me the output of "loginctl".

[Julien Aubin (gojulgarbmail)]

Hi,

When launching gdbus, error is :
julien@pcathlon64:~$ gdbus call -y -d org.freedesktop.login1 -o /org/freedesktop/login1 -m org.freedesktop.login1.Manager.CanReboot
Erreur : GDBus.Error:org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid

When trying to reboot :
julien@pcathlon64:~$ gdbus call -y -d org.freedesktop.login1 -o /org/freedesktop/login1 -m org.freedesktop.login1.Manager.Reboot true
Erreur : GDBus.Error:org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid

julien@pcathlon64:~$ loginctl
Failed to issue method call: Cannot launch daemon, file not found or permissions invalid

Actually I had a similar bug with Saucy which has been fixed with an update of logind (now replaced with systemd)

[Martin Pitt (pitti)]

If you run

  sudo /lib/systemd/systemd-logind

what happens? I suppose this crashes somehow? If not, press Control-C to end it again. Please copy&paste the output.

> logind (now replaced with systemd)

For the record, that's wrong. ConsoleKit has been replaced with logind, and we don't use systemd (in the sense of "pid 1") yet.

[Julien Aubin (gojulgarbmail)]

Oh and from my lightdm logs I also get the following messages :

QInotifyFileSystemWatcherEngine::addPaths: inotify_add_watch failed: Aucun fichier ou dossier de ce type
QFileSystemWatcher: failed to add paths: /var/lib/lightdm/.config/ibus/bus
Bus::open: Can not get ibus-daemon's address.
IBusInputContext::createInputContext: no connection to ibus-daemon
QDBusError("org.freedesktop.DBus.Error.Spawn.ExecFailed", "Cannot launch daemon, file not found or permissions invalid")
QDBusError("org.freedesktop.DBus.Error.Spawn.ExecFailed", "Cannot launch daemon, file not found or permissions invalid")
QDBusError("org.freedesktop.DBus.Error.Spawn.ExecFailed", "Cannot launch daemon, file not found or permissions invalid")
QDBusError("org.freedesktop.DBus.Error.Spawn.ExecFailed", "Cannot launch daemon, file not found or permissions invalid")

Don't know if it is related...

[Julien Aubin (gojulgarbmail)]

Hi,

For the output, it gives me the following :
administrateur@pcathlon64:~$ sudo /lib/systemd/systemd-logind
[sudo] password for administrateur:
Watching system buttons on /dev/input/event1 (Power Button)
Watching system buttons on /dev/input/event0 (Power Button)
New seat seat0.
Failed to save seat data for seat0: No such file or directory

However it does not crash...

[Martin Pitt (pitti)]

> Failed to save seat data for seat0: No such file or directory

Interesting. But the mere fact that this worked and got so far shows that logind isn't running by default, which it should. Can you please do

   sudo strace -fvvs1024 -o /tmp/logind.txt /lib/systemd/systemd-logind

and attach /tmp/logind.txt? This should give a better idea about the "Failed to save seat data for seat0".

[Julien Aubin (gojulgarbmail)]

Attached file.

[Julien Aubin (gojulgarbmail)]

I've also found this in my syslog :
Mar 26 07:46:00 pcathlon64 dbus[1249]: [system] Activating service name='org.freedesktop.login1' (using servicehelper)
Mar 26 07:46:00 pcathlon64 dbus[1249]: [system] Activated service 'org.freedesktop.login1' failed: Cannot launch daemon, file not found or permissions invalid

[Julien Aubin (gojulgarbmail)]

Also tried with qdbus with the following line (as I'm using KDE) :
qdbus --system org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager.CanReboot
Error: org.freedesktop.DBus.Error.Spawn.ExecFailed
Cannot launch daemon, file not found or permissions invalid

[Julien Aubin (gojulgarbmail)]

Another stuff in my auth.log which may help :
Mar 26 21:13:14 pcathlon64 sudo: PAM unable to dlopen(pam_cap.so): /lib/security/pam_cap.so: Ne peut ouvrir le fichier d'objet p
artagé: Aucun fichier ou dossier de ce type
Mar 26 21:13:14 pcathlon64 sudo: PAM adding faulty module: pam_cap.so

-- On every login

[Julien Aubin (gojulgarbmail)]

After installing pamcap in 64 bits message "PAM unable to dlopen(pam_cap.so): /lib/security/pam_cap.so" disappeared, however the issue is still there. :-(

[Martin Pitt (pitti)]

> 5260 mkdir("/run/systemd/seats", 0755) = -1 ENOENT (No such file or directory)

That's worrying. That means that the /etc/init/systemd-logind.conf upstart job failed somehow.

 * Do you have the libpam-systemd package installed? If that's missing, nothing will work really.

 * If you do "sudo start systemd-logind", what does that say?

[Julien Aubin (gojulgarbmail)]

Hi,

To answer your questions :
1/
julien@pcathlon64:~$ aptitude search libpam-systemd
i libpam-systemd - system and service manager - PAM module
i libpam-systemd:i386 - system and service manager - PAM module
(I tried to reinstall it, does not fix the problem)

2/
administrateur@pcathlon64:~$ sudo start systemd-logind
[sudo] password for administrateur:
systemd-logind stop/pre-start, process 4754

And when I do then ps aux|grep 4754, no process is returned. In my syslog, I get the following message :
Mar 27 08:08:09 pcathlon64 dbus[1280]: [system] Activating service name='org.freedesktop.login1' (using servicehelper)
Mar 27 08:08:09 pcathlon64 dbus[1280]: [system] Activated service 'org.freedesktop.login1' failed: Cannot launch daemon, file not found or permissions invalid

OK I've found the issue :
1/ remove libpam-systemd:i386 and libsystemd-login0:i386

Now everything should be fine.

[Julien Aubin (gojulgarbmail)]

However with the update (and a restart) I've seen that the suspend/restart/shutdown icons are still grayed out in my lightdm, and that the suspend menu is still missing from my KDE, but things are better

[Martin Pitt (pitti)]

Do I get that right, you have an amd64 system, but had libpam-systemd:i386 installed, and that confused PAM? Hmm, I wonder how this can be avoided.

With the right libpam-systemd installed, do you still get that "Cannot launch daemon" error? What does the upstart job say now (i. e. sudo start systemd-logind)?

[Julien Aubin (gojulgarbmail)]

Hi,

I had both libpam-systemd:i386 and libpam-sytemd:amd64 on my AMD64 system and this confused PAM.

The upstart job now works fine, I don't have the "cannot start daemon". I didn't post the result of "sudo start systemd-logind" but it said "started daemon systemd-logind" or something like this. And just after that reboot and shutdown menu entries appeared in KDE but not the suspend one.

The only wrong things remaining is that I cannot suspend from the GUI (but restart and shutdown appear in the KDE menu) and from lightdm suspend/reboot/shutdown entries are still disabled even after rebooting, restarting lightdm or reinstalling it.

[Julien Aubin (gojulgarbmail)]

To avoid this, it might be possible to set some kind of conflict between the i386 module and the amd64 module ? OK it smells, but this might be an option. What do you think ?

[Julien Aubin (gojulgarbmail)]

Hi,

I've also fixed the issue for the suspend feature and lightdm feature by removing packages libpam-modules:i386 and libpam-cap:i386, which have the same kind of conflict as explained above. I don't know what happened during the upgrade, but it looks like the upgrade tool was a bit lost between i386 and amd64 packages.

[Martin Pitt (pitti)]

The various PAM packages explicitly declare themselves as multi-arch compatible. Of course they could just drop their Multi-Arch: flag again so that you can only have the native PAM stack installed, not a foreign arch one in parallel. However, I suppose there was a specific reason to make them multi-arch compatible. Steve, do you know why this is beneficial, and how we could debug what's going on here if you install both stacks at the same time? Thanks!

[Alberto Salvia Novella (es20490446e)]

It has a severe impact on a large portion of Ubuntu users.

[Steve Langasek (vorlon)]

> I had both libpam-systemd:i386 and libpam-systemd:amd64
> on my AMD64 system and this confused PAM.

That makes no sense at all. PAM does not get confused by having the module available for multiple architectures. The amd64 libpam doesn't even *know* that the i386 pam module is installed, because these modules are (sensibly) each installed in architecture-specific paths which are invisible to one another.

I don't believe that having libpam-systemd:i386 installed on your system had anything to do with the system being broken, any more than libsystemd-login0:i386 did.

The only thing that removing libpam-systemd:i386 from your system does is trigger the code in the prerm:

if [ "$1" = "remove" ] ; then
    pam-auth-update --package --remove systemd
fi

However, this is a *bug* in the libpam-systemd package, because as a multiarch package, it should *not* remove the module from the config until the package is removed for all architectures. (Cf. libpam-cracklib's prerm in trusty.)

So instead of pam being confused by having libpam-systemd:i386 installed, you've broken your pam config by removing libpam-systemd:i386. And in the process, you say that the system started to work.

Without making any further changes to the packages installed, please log out and log back in to confirm that your logind session is still in a working state; and if it is, please attach the /etc/pam.d/common-session from the system.

Then please run 'sudo pam-auth-update --package', log out, log back in again, and tell us whether the suspend/reboot/shutdown options are broken.

[Julien Aubin (gojulgarbmail)]

Hi,

I've logged out and logged in and my logind session is still in a working state.

I've run sudo pam-auth-update --package and everything is fine.

Attached file /etc/pam.d/common-session

[Steve Langasek (vorlon)]

I asked that you attach the /etc/pam.d/common-session before running 'sudo pam-auth-update --package'. It appears that you've attached it after running this command, which means the file was overwritten and we can't see what state it was in before.

The file you've attached looks correct (it has pam_systemd listed in it), but this is exactly what we would expect to see after running 'pam-auth-update --package', so doesn't tell us anything.

For the sake of argument, could you please reinstall libpam-systemd:i386 and verify whether this impacts your session? It should not, but let's just rule it out completely.

If the bug does not reappear when installing libpam-systemd:i386, I'm not sure where to go with this report, as your system is surely no longer in a state that will let us reproduce the original problem; and if we can't reproduce it we probably won't be able to diagnose it.

[Julien Aubin (gojulgarbmail)]

Hi,

After reinstalling libpam-systemd:i386 systemd-logind does not start anymore, however if I remove it it starts again.

So the issue really lies within the mix libpam:i386 and libpam:amd64

[Julien Aubin (gojulgarbmail)]

OK so after rebooting I got the issue with the suspend feature. I attach you the updated common-session file

[Julien Aubin (gojulgarbmail)]

And after re-running sudo pam auto-update --package everything was OK again

[Steve Langasek (vorlon)]

Ok, this definitely shows that common-session is broken after installing libpam-systemd:i386. This confirms what you've said, but is really quite bizarre since there is nothing in the libpam-systemd maintainer scripts which should cause this.

Can you please also attach /var/log/dpkg.log and /var/log/apt/term.log from your system, so we can see what the package manager has to say about what was happening when you installed/removed these packages? (Ideally only /var/log/apt/term.log is needed, but I don't know if you were using apt to manage these packages or if you used dpkg directly)

[Steve Langasek (vorlon)]

FWIW I've tested here, and installing libpam-systemd:i386 does not break common-session for me.

[Julien Aubin (gojulgarbmail)]

Hi,

I was using aptitude to manage packages...

[Julien Aubin (gojulgarbmail)]

Attached /var/log/dpkg.log

[Steve Langasek (vorlon)]

Nothing obviously wrong in the logs. term.log shows:

Log started: 2014-03-28 22:16:57
dpkg : avertissement : analyse du fichier '/var/lib/dpkg/available' vers la ligne 1237409 paquet 'yabause-qt' :
 champ « Depends », référence à « libqt4-opengl » : erreur dans la version: caractère non valable dans le numéro de version
Sélection du paquet libpam-systemd:i386 précédemment désélectionné.
(Lecture de la base de données... ^M(Lecture de la base de données... 5%^M(Lecture de la base de données... 10%^M(Lecture de la base de données... 15%^M(Lecture de la base de données... 20%^M(Lecture de la base de données... 25%^M(Lecture de la base de données... 30%^M(Lecture de la base de données... 35%^M(Lecture de la base de données... 40%^M(Lecture de la base de données... 45%^M(Lecture de la base de données... 50%^M(Lecture de la base de données... 55%^M(Lecture de la base de données... 60%^M(Lecture de la base de données... 65%^M(Lecture de la base de données... 70%^M(Lecture de la base de données... 75%^M(Lecture de la base de données... 80%^M(Lecture de la base de données... 85%^M(Lecture de la base de données... 90%^M(Lecture de la base de données... 95%^M(Lecture de la base de données... 100%^M(Lecture de la base de données... 339549 fichiers et répertoires déjà installés.)
Preparing to unpack .../libpam-systemd_204-5ubuntu13_i386.deb ...
Unpacking libpam-systemd:i386 (204-5ubuntu13) ...
Processing triggers for ureadahead (0.100.0-16) ...
Processing triggers for man-db (2.6.6-1) ...
Paramétrage de libpam-systemd:i386 (204-5ubuntu13) ...
Log ended: 2014-03-28 22:17:20

No other packages being changed in unexpected ways, here, and the configuration of the package seems to have proceeded normally.

There is a dpkg error message about 'yabause-qt' that you should probably follow up on; this is a warning now but may become an error in the future, so whatever this unofficial version of yabause-qt is, you probably want to remove it from your sources.

But I don't see how that explains the issue you're seeing.

Is there any chance that you have a locally-modified version of pam-auth-update installed? Can you install the 'debsums' package and post the output of 'sudo debsums -s'?

[Steve Langasek (vorlon)]

and addition to the 'debsums' output, can you please confirm that 'which pam-auth-update' returns '/usr/sbin/pam-auth-update'

[Julien Aubin (gojulgarbmail)]

which pam-auth-update :
/usr/sbin/pam-auth-update

For the yabause-qt it's an error in /var/lib/dpkg/available, nothing unofficial. I've removed it from /var/lib/dpkg/available and the warning is gone. Probably some encoding issue.

Output of my debsums (some errors but nothing looks really critical) :

administrateur@pcathlon64:~$ sudo debsums -s
[sudo] password for administrateur:
debsums: invalid line (1) in md5sums for devolo-dlan-cockpit: /mnt/repository/kolja/dune/software-packages/cockpit/obj.linux.rel/tmp.DwmJybAOOr/debian /mnt/repository/kolja/dune/software-packages/cockpit/source.linux
debsums: missing file /usr/lib/i386-linux-gnu/libgphoto2/print-camera-list (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/AUTHORS (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/NEWS.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/README.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.9050 (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.9051.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.9052 (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.905C.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.913C.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.913D (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.adc65 (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.agfa-cl20 (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.aox.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.ax203-compression.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.ax203.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.canon.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.clicksmart310 (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.enigma13 (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.gsmart300 (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.iclick.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.jamcam (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.jl2005a.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.jl2005bcd-compression.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camlibs/README.jl2005c.gz (from libgphoto2-2:i386 package)
debsums: missing file /usr/share/doc/libgphoto2-2/camli...

[Steve Langasek (vorlon)]

On Sun, Mar 30, 2014 at 07:02:47PM -0000, Julien Aubin wrote:
> Output of my debsums (some errors but nothing looks really critical) :

Ok. I'm sorry to say that at this point, I have no other ideas. Something
is going wrong with the install of this package on your system, but I can't
reproduce it here and there are no indications of any problems in your logs.

[Julien Aubin (gojulgarbmail)]

Once I've removed libgphoto2-2 :
^Cadministrateur@pcathlon64:~$ sudo debsums -s
[sudo] password for administrateur:
debsums: invalid line (1) in md5sums for devolo-dlan-cockpit: /mnt/repository/kolja/dune/software-packages/cockpit/obj.linux.rel/tmp.DwmJybAOOr/debian /mnt/repository/kolja/dune/software-packages/cockpit/source.linux

[Gavin Graham (gavingraham)]

Since updating this morning, my Trusty install is now showing:

Apr 4 17:03:26 Gavin dbus[848]: [system] Activated service 'org.freedesktop.login1' failed: Cannot launch daemon, file not found or permissions invalid

libpam-systemd-204-5ubuntu16

Also can't mount external drives and anything else that depends on policykit. Can unlock admin under settings either although I am in the 'adm' group.

For example, trying to run update-manager yields:
org.freedesktop.PolicyKit.Error.Failed: ('system-bus-name', {'name': ':1.178'}): org.debian.apt.update-cache

[unknown (kirokko-deactivatedaccount)]

I've got the same problem as Gavin Gragam described earlier. I'm using Xubuntu 14.04 for a week, everything works fine until today. After last upgrade I've got a huge problem. I cannot mount external USB NTFS Driver via Thunar and cannot use shutdown, restart neither in system menu or login screen.

[John Franklin (ubuntu-sentaidigital)]

@gavingraham and @kirkko, you may be suffering from this bug. https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1302264

[Gavin Graham (gavingraham)]

I was indeed suffering from https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1302264
They have patched systemd-services and things are working again for me.

[Alberto Salvia Novella (es20490446e)]

It has a severe impact on a large portion of Ubuntu users.

[Steve Langasek (vorlon)]

If the bug were reproducible, critical would probably be the right severity. But so far, it's only seen on the system of a single user and we don't know why.

[Paul Andreassen (paulone)]

I have a pre 12.04LTS Kubuntu system I've upgrade to 12.04LTS and now to 14.04LTS. I remember the unable to logout, shutdown etc from 12.04 and the solution that worked then and now for me on 14.04LTS was:

sudo apt-get install --reinstall polkit-kde-1

and then reboot.

[Assarte D'Raven (fejlesztes-szikra)]

Hi,

Same effects on here after installing an IM named "Brosix" from http://download.brosix.com/get/?nid=1&fid=Brosix_ubuntu ...or some update, I don't sure but Brosix installed many i386 packages. After yesterday I have permission issues clearly, I cannot use shutdown, suspend, restart on lightdm nor KDE from tray or K-launch menu, I cannot mount partitions from Dolphin, admin operations cannot be taken because auth dialog doesn't popped up. This is on an AMD64 desktop machine with Trusty 14.04, Kubuntu.

I used the commands below as Julien's suggested previously:
sudo pam auto-update --package
sudo apt-get remove libpam-systemd:i386 and libsystemd-login0:i386
sudo apt-get remove libpam-modules:i386 libpam-cap:i386

After run these commands and a restart from terminal with sodo shutdown -r now problems are disappeared but Brosix removed too (because of its dependencies, I think).

[Michael Heuberger (michael.heuberger)]

Not sure if my problem is related to this one. At the moment I experience issues installing/updating libpam-systemd:amd64. Then I followed the installation scripts and saw that /sys/fs/cgroup/systemd is missing and I wasn't able to create that manually or using the script. Looks like /sys/fs/cgroup is missing.

Sorry if this is off-topic but I am desperate to fix it. Here a log from sudo apt-get install libpam-systemd:

...
Setting up libpam-systemd:amd64 (204-5ubuntu20.4) ...
start: Job failed to start
invoke-rc.d: initscript systemd-logind, action "start" failed.
dpkg: error processing package libpam-systemd:amd64 (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 libpam-systemd:amd64

[Martin Pitt (pitti)]

Michael, sounds like you are running a custom kernel without cgroup support? (see bug 1325309). Fix it by enabling cgroups in the kernel.

[Martin Pitt (pitti)]

Julien et al, please have a look at bug 1382510. This is currently the most obvious case of logind breakage with libpam-systemd for both i386 and amd64 installed. (I. e. this bug is very likely a duplicate of 1382510)