ubiquity and others time out on polkit (killed by udisks2-inhibit)

Bug #1508075 reported by Jonathan Riddell on 2015-10-20
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
udisks2 (Ubuntu)
Critical
Martin Pitt
Wily
Critical
Martin Pitt
Xenial
Medium
Martin Pitt

Bug Description

on wily release candidates I get a crash when it switches to the network setup page

this doesn't happen with "try ubuntu" mode which runs as root only when launching ubiquity from the desktop

the backtrace points to an authentication issue

I get a related backtrace even if I'm connected to a network from the panel network-manager

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: ubiquity 2.21.35
ProcVersionSignature: Ubuntu 4.2.0-16.19-generic 4.2.3
Uname: Linux 4.2.0-16-generic x86_64
ApportVersion: 2.19.1-0ubuntu3
Architecture: amd64
CasperVersion: 1.365
CurrentDesktop: Unity
Date: Tue Oct 20 14:26:28 2015
InstallCmdLine: file=/cdrom/preseed/ubuntu.seed boot=casper initrd=/casper/initrd.lz quiet splash ---
LiveMediaBuild: Ubuntu 15.10 "Wily Werewolf" - Release amd64 (20151019)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: ubiquity
UpgradeStatus: No upgrade log present (probably fresh install)

Jonathan Riddell (jr) wrote :
Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1508075

tags: added: iso-testing

Seems to be broken maybe in policykit-1: when you try to start ubiquity, it appears as if polkitd gets DBus activated, then stopped when ubiquity stops, but this fails if it doesn't manage to get activated correctly (or works more than once if it fails to stop), etc.

This needs to be looked at carefully in DBus activation and systemd configs to make sure there isn't some issue with how it gets started in the live session. Basically, crashes only happen if polkit isn't running when ubiquity starts.

affects: ubiquity (Ubuntu) → policykit-1 (Ubuntu)
Changed in policykit-1 (Ubuntu):
importance: Undecided → Critical
status: New → Confirmed
Martin Pitt (pitti) wrote :

I tried today's ISO: Ubuntu 15.10 "Wily Werewolf" - Release amd64 (20151020)

On my UEFI laptop I don't get the graphical "Try Ubuntu / Install Ubuntu" picker, just the grub one. I tried all four cases of "try" (live session) and "install" (ubiquity-only) and connect via indicator vs. connect in ubiquity, and it worked fine. Also, I checked (in separate reboots, to not influence the first four tests before) that in all of these cases polkitd was already running before I did anything (switched to VT1).

I then tried the same iso on my wife's laptop with BIOS, and do get the graphical install vs. try picker. I can connect there via the indicator as well. I also booted it in QEMU, and polkit starts up properly (naturally I can't verify wifi there).

So I cannot reproduce this myself.

In the syslog it appears that polkit started up fine:

Oct 20 14:21:06 ubuntu dbus[1414]: [system] Activating via systemd: service name='org.freedesktop.PolicyKit1' unit='polkitd.service'
Oct 20 14:21:06 ubuntu systemd[1]: Started Authenticate and Authorize Users to Run Privileged Tasks.

which means that polkit's bus name was properly claimed (as that's what systemd considers as "started" for Type=dbus).

but 25s later it timed out, which is exactly the default D-Bus timeout:

Oct 20 14:21:31 ubuntu NetworkManager[1373]: <warn> error requesting auth for org.freedesktop.NetworkManager.wifi.share.protected: (0) Authorization check failed: GDBus.Error:org.freedesktop.DBus.Error.TimedOut: Failed to activate service 'org.freedesktop.PolicyKit1': timed out

so it seems it registered on the bus but then froze (I don't think it crashed as that should appear in the logs somewhere.

Can you please boot the live system with "break=casper-bottom", and run

   sed -i '/^Exec/ s!=!=/usr/bin/strace -fvvs1024 -o /run/polkit.trace !' /root/lib/systemd/system/polkitd.service

(no editor on the initramfs..) then continue booting with Ctrl+D, then go up to the point when polkit freezes. Switch to VT1, log in (user ubuntu, no password), and salvage /run/polkit.trace. I tested these steps in a VM, but of course my polkit.trace doesn't tell me anything interesting.

Thanks!

Changed in policykit-1 (Ubuntu):
status: Confirmed → Incomplete
Martin Pitt (pitti) wrote :

Thanks Mathieu; all your traces indeed have

   +++ killed by SIGHUP +++

which is from /usr/lib/udisks2/udisks2-inhibit. This is a race condition, when polkit gets killed at the wrong time while a request is pending it would behave that way. Curious that this only surfaces right now, /usr/lib/udisks2/udisks2-inhibit has been like that for a while already.

All of your traces got interrupted in poll() though, i. e. there was no pending request (no recv* yet), so this did not interrupt a pending call. However, this is still bad.

Thing to try on a machine which reproduces this: Please comment out all the code except the final "$@" from /usr/lib/udisks2/udisks2-inhibit (from initramfs break=casper-bottom) and verify that this crash then stops. Then, once you are in ubiquity, run "pkill -e --signal STOP gvfs-udisks" to stop gvfs-udisks2-volume-monitor (but that does not apply to all derivatives, such as KDE).

I'll think about how we can inhibit udisks without having to restart polkit.

Martin Pitt (pitti) wrote :

Another thing to try: Replace

   pkill --signal HUP polkitd

with

   systemctl try-restart polkitd.service

in /usr/lib/udisks2/udisks2-inhibit. That will immediately restart it, instead of waiting for another d-bus activation cycle (which might confuse existing clients).

affects: policykit-1 (Ubuntu) → udisks2 (Ubuntu)
summary: - crash during network setup in ubiquity wily
+ ubiquity and others time out on polkit (killed by udisks2-inhibit)
Martin Pitt (pitti) on 2015-10-21
Changed in udisks2 (Ubuntu):
status: Incomplete → In Progress
assignee: nobody → Martin Pitt (pitti)
Martin Pitt (pitti) wrote :

This is a tested replacement for udisks2-inhibit which avoids killing/restarting the daemon. It instead uses polkit's inotify by merely placing a symlink into /var/lib/polkit-1/localauthority/90-mandatory.d/ . Since we are only adding a new file (or rm it on cleanup), overlayfs' restricted inotify support actually seems good enough.

You can test quickly with

  pkcheck -a org.freedesktop.udisks2.filesystem-mount -p $$; echo $?

This should give 0 ("allowed") normally, and 1 and "Not authorized." while you are running "sudo ./udisks2-inhibit sleep 5" or something such.

Martin Pitt (pitti) wrote :

Note that the above is less robust than the original one as it now could happen to leave a dangling symlink behind if udisks2-inhibit is killed while its running with a signal that doesn't run the cleanup trap. However, this doesn't change actual policy, and as we only really use that during the live system (where such leftover files don't matter at all), it'd be acceptable.

The bind mount to /run didn't have this problem, but it doesn't trigger inotify and thus requires a restart (which is problematic).

Martin Pitt (pitti) wrote :

As indicated above, this is a variant which keeps the robust bind mounting, but restarts polkit instead of just killing it. This should deal much more gracefully with existing connected clients.

Martin Pitt (pitti) wrote :

Neither of the above solutions are perfect, thus for x and beyond (and for Debian), I'd actually like to change it like this: udisks2-inhibit should create a temporary udev rule in /run/udev/rules.d/ with

  SUBSYSTEM=="block", ENV{UDISKS_IGNORE}="1" on all block devices.

and then "udevadm trigger -subsystem-match=block" to apply it to existing devices.

Note that UDISKS_AUTO="0" won't suffice: We don't only want to avoid auto-mounting external USB storage (as we might install onto that), but also avoid accidental manual mounts by clicking on newly appearing drive icons.

Changed in udisks2 (Ubuntu Wily):
status: In Progress → Fix Committed
Martin Pitt (pitti) wrote :

For Debian and X-series I have committed http://anonscm.debian.org/cgit/pkg-utopia/udisks2.git/commit/?id=b68ac3766c now, which implements the udev rules approach.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package udisks2 - 2.1.6-2ubuntu1

---------------
udisks2 (2.1.6-2ubuntu1) wily; urgency=medium

  * udisks2-inhibit: Restart polkitd instead of just killing it, so that
    existing connected clients continue having something to talk to instead of
    timing out. Keep old pkill as a fallback for non-systemd.
    (LP: #1508075)

 -- Martin Pitt <email address hidden> Wed, 21 Oct 2015 13:41:35 +0200

Changed in udisks2 (Ubuntu Wily):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package udisks2 - 2.1.6-2+git1

---------------
udisks2 (2.1.6-2+git1) xenial; urgency=medium

  * udisks2-inhibit: Stop fiddling with polkit rules; restarting it can break
    existing clients/pending requests, and we can't use inotify as we don't
    want to write anything on the actual file system and bind mounts don't
    trigger inotify. Use a different approach of a temporary udev rule which
    sets UDISKS_IGNORE on all block devices. This continues to avoid touching
    the file system but does not need any daemon restarts, works with polkit
    >= 106, and now also suppresses showing new block devices on the desktop.
    (LP: #1508075)

 -- Martin Pitt <email address hidden> Thu, 29 Oct 2015 12:04:08 +0100

Changed in udisks2 (Ubuntu Xenial):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers