snapd remove /usr/local/bin from the PATH for all systemd unit (bionic SRU regression)

Bug #1814355 reported by E. MAS on 2019-02-02
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
initramfs-tools (Ubuntu)
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned
snapd (Ubuntu)
High
Unassigned
Bionic
Critical
Unassigned
Cosmic
Undecided
Unassigned
systemd (Ubuntu)
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned

Bug Description

[Impact]

 * Initramfs exports PATH to init, which is different than the expected stock / compiled one, which results in slightly different runtime behaviour of init, if it has environment generators as well.

[Test Case]

 * Disable snapd env generator & disable initrd-less boot (if enabled)
   sudo chmod -x /usr/lib/systemd/system-environment-generators/snapd-env-generator
   set empty GRUB_FORCE_PARTUUID= and update-grub

 * Reboot cosmic system with an initramfs
   $ journalctl -b -k | grep initramfs
   (verify that initramfs was unpacked)

 * Check the path used by systemd, ie.:
   systemd-run /usr/bin/env
   journalctl -b -e | grep PATH

   It should contain /usr/local

 * Enable any env generator for example & reboot:
   /usr/lib/systemd/system-environment-generators/xnox.sh
   #!/bin/sh
   echo XNOX=ROCKS

 * Verify path used by systemd

   It should still contain /usr/local

Repeat again with the new initramfs-tools.

[Regression Potential]

 * We are hardcoding, the same path, yet again, in one more place. However, we are setting it to a well-known value (see https://wiki.ubuntu.com/PATH
 ) as it is universally expected and regressed when booted (a) with initrd AND (b) with broken path exported by initrd AND (c) with an env-generator.

[Other Info]

 * Anything else you think is useful to include
 * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
 * and address these questions in advance

===

Big regression in 2.37.1+18.04 compare to version 2.34.2

all these paths /usr/local/sbin & /usr/local/bin are not anymore in the path of all systemd process .

So we can not start a daemon that use /usr/local/bin

reinstalling package 2.34.2 fix the problem

in 2.34.2 :

~# strings /proc/$(pidof /lib/systemd/systemd-resolved)/environ | grep PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

in 2.37.1+18.04 :

~# strings /proc/$(pidof /lib/systemd/systemd-resolved)/environ | grep PATH
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/snap/bin

tags: added: regression-update
Oliver Grawert (ogra) wrote :

likely fallout of the fix from 1771858

Changed in snapd (Ubuntu):
importance: Undecided → High
summary: - snapd remove /usr/local/bin from the PATH for all systemd unit
+ snapd remove /usr/local/bin from the PATH for all systemd unit (bionic
+ SRU regression)
E. MAS (erwan-mas) wrote :

This is not a duplicate of 1771858 . This a critical regression from update for fixing 1771858 .

Since we have the bug we can not deploy anymoore kubernetes charm fro canonical .

E. MAS (erwan-mas) wrote :

This is related to 1771858 , i agree .

Alexander Turek (turekaj) wrote :

agree, cannot deploy Kubernetes charm from canonical, or add Kubernetes-worker units to existing deployment

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in snapd (Ubuntu):
status: New → Confirmed
Dimitri John Ledkov (xnox) wrote :

Hi,

I cannot reproduce the issue, but it might be relevant how the system is booted and/or was upgraded.

Can you do the following, please?

sudo systemd-run --unit=check-env /usr/bin/env
sudo journalctl --no-pager -u check-env
cat /etc/systemd/system.conf
cat /proc/cmdline
journalctl -b -k | grep -e initrd -e command
ls /usr/lib/systemd/system-environment-generators/

And paste the output here?

Michael Vogt (mvo) wrote :

Here is what I see in a clean VM:

ubuntu@autopkgtest:~$ apt list snapd
Listing... Done
snapd/bionic-updates 2.33.1+18.04ubuntu2 amd64
N: There is 1 additional version. Please use the '-a' switch to see it

ubuntu@autopkgtest:~$ sudo systemd-run --pty --wait '/bin/echo' '$PATH'
Running as unit: run-u7.service
Press ^] three times within 1s to disconnect TTY.
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 13ms

After just upgrading snapd (latest version) and rebooting:

ubuntu@autopkgtest:~$ apt list snapd
Listing... Done
snapd/bionic-updates,now 2.37.1+18.04 amd64 [installed]

ubuntu@autopkgtest:~$ sudo systemd-run --pty --wait '/bin/echo' '$PATH'
Running as unit: run-u7.service
Press ^] three times within 1s to disconnect TTY.
/sbin:/usr/sbin:/bin:/usr/bin:/snap/bin
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 15ms

So this appears to be real. We added the systemd environment generator (/usr/lib/systemd/system-environment-generators/snapd-env-generator) in 2.35.1 so that seems like a likely culprit. We also ship this in 18.10 where it does not have these ill effects (but there is also a different systemd version used there).

Michael Vogt (mvo) wrote :

As for the questions of xnox:
"""
ubuntu@autopkgtest:~$ sudo systemd-run --unit=check-env /usr/bin/env
Running as unit: check-env.service
ubuntu@autopkgtest:~$ sudo journalctl --no-pager -u check-env
-- Logs begin at Mon 2018-07-23 18:30:47 CEST, end at Sun 2019-02-03 13:25:21 CET. --
Feb 03 13:25:21 autopkgtest systemd[1]: Started /usr/bin/env.
Feb 03 13:25:21 autopkgtest env[543]: LANG=en_US.UTF-8
Feb 03 13:25:21 autopkgtest env[543]: PATH=/sbin:/usr/sbin:/bin:/usr/bin:/snap/bin
Feb 03 13:25:21 autopkgtest env[543]: INVOCATION_ID=4c9b05e577224d6284cc641778807a16
Feb 03 13:25:21 autopkgtest env[543]: JOURNAL_STREAM=9:18437
ubuntu@autopkgtest:~$ cat /etc/systemd/system.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See systemd-system.conf(5) for details.

[Manager]
#LogLevel=info
#LogTarget=journal-or-kmsg
#LogColor=yes
#LogLocation=no
#DumpCore=yes
#ShowStatus=yes
#CrashChangeVT=no
#CrashShell=no
#CrashReboot=no
#CtrlAltDelBurstAction=reboot-force
#CPUAffinity=1 2
#JoinControllers=cpu,cpuacct net_cls,net_prio
#RuntimeWatchdogSec=0
#ShutdownWatchdogSec=10min
#CapabilityBoundingSet=
#SystemCallArchitectures=
#TimerSlackNSec=
#DefaultTimerAccuracySec=1min
#DefaultStandardOutput=journal
#DefaultStandardError=inherit
#DefaultTimeoutStartSec=90s
#DefaultTimeoutStopSec=90s
#DefaultRestartSec=100ms
#DefaultStartLimitIntervalSec=10s
#DefaultStartLimitBurst=5
#DefaultEnvironment=
#DefaultCPUAccounting=no
#DefaultIOAccounting=no
#DefaultIPAccounting=no
#DefaultBlockIOAccounting=no
#DefaultMemoryAccounting=no
#DefaultTasksAccounting=yes
#DefaultTasksMax=
#DefaultLimitCPU=
#DefaultLimitFSIZE=
#DefaultLimitDATA=
#DefaultLimitSTACK=
#DefaultLimitCORE=
#DefaultLimitRSS=
#DefaultLimitNOFILE=
#DefaultLimitAS=
#DefaultLimitNPROC=
#DefaultLimitMEMLOCK=
#DefaultLimitLOCKS=
#DefaultLimitSIGPENDING=
#DefaultLimitMSGQUEUE=
#DefaultLimitNICE=
#DefaultLimitRTPRIO=
#DefaultLimitRTTIME=
#IPAddressAllow=
#IPAddressDeny=
ubuntu@autopkgtest:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-45-generic root=UUID=991dab44-ecab-46c0-be75-150dbea660be ro console=ttyS0
ubuntu@autopkgtest:~$ journalctl -b -k | grep -e initrd -e command
Feb 03 13:23:52 autopkgtest kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-45-generic root=UUID=991dab44-ecab-46c0-be75-150dbea660be ro console=ttyS0
Feb 03 13:23:52 autopkgtest kernel: Freeing initrd memory: 50848K
ubuntu@autopkgtest:~$ ls /usr/lib/systemd/system-environment-generators/
snapd-env-generator
"""

Changed in snapd (Ubuntu Bionic):
importance: Undecided → Critical

Hello E., or anyone else affected,

Accepted snapd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snapd/2.37.1.1+18.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in snapd (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-bionic
Will Cooke (willcooke) wrote :

Performing the same tests as mvo. Before upgrading...

$ apt list snapd
Listing... Done
snapd/bionic-updates 2.37.1+18.04 amd64 [upgradable from: 2.35.5+18.04]
N: There are 2 additional versions. Please use the '-a' switch to see them.

$ sudo systemd-run --pty --wait '/bin/echo' '$PATH'
Running as unit: run-u920.service
Press ^] three times within 1s to disconnect TTY.
/sbin:/usr/sbin:/bin:/usr/bin:/snap/bin
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 10ms

Will Cooke (willcooke) wrote :

After upgrade:

$ apt list snapd
Listing... Done
snapd/bionic-updates,now 2.37.1+18.04 amd64 [installed]
N: There is 1 additional version. Please use the '-a' switch to see it

$ sudo systemd-run --pty --wait '/bin/echo' '$PATH'
[sudo] password for will:
Running as unit: run-u121.service
Press ^] three times within 1s to disconnect TTY.
/sbin:/usr/sbin:/bin:/usr/bin:/snap/bin
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 8ms

Will Cooke (willcooke) wrote :

Ignore the previous comment, I still had the wrong version installed. Let's try that again...

$ apt list snapd
Listing... Done
snapd/bionic-proposed,now 2.37.1.1+18.04 amd64 [installed]
N: There are 2 additional versions. Please use the '-a' switch to see them.

$ sudo systemd-run --pty --wait '/bin/echo' '$PATH'
[sudo] password for will:
Running as unit: run-u120.service
Press ^] three times within 1s to disconnect TTY.
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 13ms

tags: added: verification-done-bionic
removed: verification-needed-bionic
tags: added: verification-done
removed: verification-needed
Will Cooke (willcooke) wrote :

The path is correct again. Marking as verification-done.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package snapd - 2.37.1.1+18.04

---------------
snapd (2.37.1.1+18.04) bionic; urgency=medium

  * New upstream release, LP: #1811233
    - disable systemd environment generator on bionic to fix
      LP: #1814355

 -- Michael Vogt <email address hidden> Sun, 03 Feb 2019 15:20:57 +0100

Changed in snapd (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for snapd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

E. MAS (erwan-mas) wrote :

The release of 2.37.1.1 fixed my issue .

Dimitri John Ledkov (xnox) wrote :

Also initramfs-tools sets PATH without /usr/local, which may leak into init environment. Fixing this as well.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package initramfs-tools - 0.131ubuntu17

---------------
initramfs-tools (0.131ubuntu17) disco; urgency=medium

  * Fix PATH to include /usr/local. LP: #1814355

 -- Dimitri John Ledkov <email address hidden> Tue, 12 Feb 2019 13:16:06 +0000

Changed in initramfs-tools (Ubuntu):
status: New → Fix Released
description: updated
description: updated
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers