ubiquity and others time out on polkit (killed by udisks2-inhibit)

[Jonathan Riddell]

on wily release candidates I get a crash when it switches to the network setup page

this doesn't happen with "try ubuntu" mode which runs as root only when launching ubiquity from the desktop

the backtrace points to an authentication issue

I get a related backtrace even if I'm connected to a network from the panel network-manager

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: ubiquity 2.21.35
ProcVersionSignature: Ubuntu 4.2.0-16.19-generic 4.2.3
Uname: Linux 4.2.0-16-generic x86_64
ApportVersion: 2.19.1-0ubuntu3
Architecture: amd64
CasperVersion: 1.365
CurrentDesktop: Unity
Date: Tue Oct 20 14:26:28 2015
InstallCmdLine: file=/cdrom/preseed/ubuntu.seed boot=casper initrd=/casper/initrd.lz quiet splash ---
LiveMediaBuild: Ubuntu 15.10 "Wily Werewolf" - Release amd64 (20151019)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: ubiquity
UpgradeStatus: No upgrade log present (probably fresh install)

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1508075

[Mathieu Trudel-Lapierre (cyphermox)]

Seems to be broken maybe in policykit-1: when you try to start ubiquity, it appears as if polkitd gets DBus activated, then stopped when ubiquity stops, but this fails if it doesn't manage to get activated correctly (or works more than once if it fails to stop), etc.

This needs to be looked at carefully in DBus activation and systemd configs to make sure there isn't some issue with how it gets started in the live session. Basically, crashes only happen if polkit isn't running when ubiquity starts.

[Martin Pitt (pitti)]

I tried today's ISO: Ubuntu 15.10 "Wily Werewolf" - Release amd64 (20151020)

On my UEFI laptop I don't get the graphical "Try Ubuntu / Install Ubuntu" picker, just the grub one. I tried all four cases of "try" (live session) and "install" (ubiquity-only) and connect via indicator vs. connect in ubiquity, and it worked fine. Also, I checked (in separate reboots, to not influence the first four tests before) that in all of these cases polkitd was already running before I did anything (switched to VT1).

I then tried the same iso on my wife's laptop with BIOS, and do get the graphical install vs. try picker. I can connect there via the indicator as well. I also booted it in QEMU, and polkit starts up properly (naturally I can't verify wifi there).

So I cannot reproduce this myself.

In the syslog it appears that polkit started up fine:

Oct 20 14:21:06 ubuntu dbus[1414]: [system] Activating via systemd: service name='org.freedesktop.PolicyKit1' unit='polkitd.service'
Oct 20 14:21:06 ubuntu systemd[1]: Started Authenticate and Authorize Users to Run Privileged Tasks.

which means that polkit's bus name was properly claimed (as that's what systemd considers as "started" for Type=dbus).

but 25s later it timed out, which is exactly the default D-Bus timeout:

Oct 20 14:21:31 ubuntu NetworkManager[1373]: <warn> error requesting auth for org.freedesktop.NetworkManager.wifi.share.protected: (0) Authorization check failed: GDBus.Error:org.freedesktop.DBus.Error.TimedOut: Failed to activate service 'org.freedesktop.PolicyKit1': timed out

so it seems it registered on the bus but then froze (I don't think it crashed as that should appear in the logs somewhere.

Can you please boot the live system with "break=casper-bottom", and run

   sed -i '/^Exec/ s!=!=/usr/bin/strace -fvvs1024 -o /run/polkit.trace !' /root/lib/systemd/system/polkitd.service

(no editor on the initramfs..) then continue booting with Ctrl+D, then go up to the point when polkit freezes. Switch to VT1, log in (user ubuntu, no password), and salvage /run/polkit.trace. I tested these steps in a VM, but of course my polkit.trace doesn't tell me anything interesting.

Thanks!

[Martin Pitt (pitti)]

Thanks Mathieu; all your traces indeed have

   +++ killed by SIGHUP +++

which is from /usr/lib/udisks2/udisks2-inhibit. This is a race condition, when polkit gets killed at the wrong time while a request is pending it would behave that way. Curious that this only surfaces right now, /usr/lib/udisks2/udisks2-inhibit has been like that for a while already.

All of your traces got interrupted in poll() though, i. e. there was no pending request (no recv* yet), so this did not interrupt a pending call. However, this is still bad.

Thing to try on a machine which reproduces this: Please comment out all the code except the final "$@" from /usr/lib/udisks2/udisks2-inhibit (from initramfs break=casper-bottom) and verify that this crash then stops. Then, once you are in ubiquity, run "pkill -e --signal STOP gvfs-udisks" to stop gvfs-udisks2-volume-monitor (but that does not apply to all derivatives, such as KDE).

I'll think about how we can inhibit udisks without having to restart polkit.

[Martin Pitt (pitti)]

Another thing to try: Replace

   pkill --signal HUP polkitd

with

   systemctl try-restart polkitd.service

in /usr/lib/udisks2/udisks2-inhibit. That will immediately restart it, instead of waiting for another d-bus activation cycle (which might confuse existing clients).

[Martin Pitt (pitti)]

This is a tested replacement for udisks2-inhibit which avoids killing/restarting the daemon. It instead uses polkit's inotify by merely placing a symlink into /var/lib/polkit-1/localauthority/90-mandatory.d/ . Since we are only adding a new file (or rm it on cleanup), overlayfs' restricted inotify support actually seems good enough.

You can test quickly with

  pkcheck -a org.freedesktop.udisks2.filesystem-mount -p $$; echo $?

This should give 0 ("allowed") normally, and 1 and "Not authorized." while you are running "sudo ./udisks2-inhibit sleep 5" or something such.

[Martin Pitt (pitti)]

Note that the above is less robust than the original one as it now could happen to leave a dangling symlink behind if udisks2-inhibit is killed while its running with a signal that doesn't run the cleanup trap. However, this doesn't change actual policy, and as we only really use that during the live system (where such leftover files don't matter at all), it'd be acceptable.

The bind mount to /run didn't have this problem, but it doesn't trigger inotify and thus requires a restart (which is problematic).

[Martin Pitt (pitti)]

As indicated above, this is a variant which keeps the robust bind mounting, but restarts polkit instead of just killing it. This should deal much more gracefully with existing connected clients.

[Martin Pitt (pitti)]

Neither of the above solutions are perfect, thus for x and beyond (and for Debian), I'd actually like to change it like this: udisks2-inhibit should create a temporary udev rule in /run/udev/rules.d/ with

  SUBSYSTEM=="block", ENV{UDISKS_IGNORE}="1" on all block devices.

and then "udevadm trigger -subsystem-match=block" to apply it to existing devices.

Note that UDISKS_AUTO="0" won't suffice: We don't only want to avoid auto-mounting external USB storage (as we might install onto that), but also avoid accidental manual mounts by clicking on newly appearing drive icons.

[Martin Pitt (pitti)]

For Debian and X-series I have committed http://anonscm.debian.org/cgit/pkg-utopia/udisks2.git/commit/?id=b68ac3766c now, which implements the udev rules approach.

[Launchpad Janitor (janitor)]

This bug was fixed in the package udisks2 - 2.1.6-2ubuntu1

---------------
udisks2 (2.1.6-2ubuntu1) wily; urgency=medium

  * udisks2-inhibit: Restart polkitd instead of just killing it, so that
    existing connected clients continue having something to talk to instead of
    timing out. Keep old pkill as a fallback for non-systemd.
    (LP: #1508075)

 -- Martin Pitt <email address hidden> Wed, 21 Oct 2015 13:41:35 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package udisks2 - 2.1.6-2+git1

---------------
udisks2 (2.1.6-2+git1) xenial; urgency=medium

  * udisks2-inhibit: Stop fiddling with polkit rules; restarting it can break
    existing clients/pending requests, and we can't use inotify as we don't
    want to write anything on the actual file system and bind mounts don't
    trigger inotify. Use a different approach of a temporary udev rule which
    sets UDISKS_IGNORE on all block devices. This continues to avoid touching
    the file system but does not need any daemon restarts, works with polkit
    >= 106, and now also suppresses showing new block devices on the desktop.
    (LP: #1508075)

 -- Martin Pitt <email address hidden> Thu, 29 Oct 2015 12:04:08 +0100