snapd remove /usr/local/bin from the PATH for all systemd unit (bionic SRU regression)

[E. MAS]

[Impact]

 * Initramfs exports PATH to init, which is different than the expected stock / compiled one, which results in slightly different runtime behaviour of init, if it has environment generators as well.

[Test Case]

 * Disable snapd env generator & disable initrd-less boot (if enabled)
   sudo chmod -x /usr/lib/systemd/system-environment-generators/snapd-env-generator
   set empty GRUB_FORCE_PARTUUID= and update-grub

 * Reboot cosmic system with an initramfs
   $ journalctl -b -k | grep initramfs
   (verify that initramfs was unpacked)

 * Check the path used by systemd, ie.:
   systemd-run /usr/bin/env
   journalctl -b -e | grep PATH

   It should contain /usr/local

 * Enable any env generator for example & reboot:
   /usr/lib/systemd/system-environment-generators/xnox.sh
   #!/bin/sh
   echo XNOX=ROCKS

 * Verify path used by systemd

   It should still contain /usr/local

Repeat again with the new initramfs-tools.

[Regression Potential]

 * We are hardcoding, the same path, yet again, in one more place. However, we are setting it to a well-known value (see https://wiki.ubuntu.com/PATH
 ) as it is universally expected and regressed when booted (a) with initrd AND (b) with broken path exported by initrd AND (c) with an env-generator.

[Other Info]

 * Anything else you think is useful to include
 * Anticipate questions from users, SRU, +1 maintenance, security teams and the Technical Board
 * and address these questions in advance

===

Big regression in 2.37.1+18.04 compare to version 2.34.2

all these paths /usr/local/sbin & /usr/local/bin are not anymore in the path of all systemd process .

So we can not start a daemon that use /usr/local/bin

reinstalling package 2.34.2 fix the problem

in 2.34.2 :

~# strings /proc/$(pidof /lib/systemd/systemd-resolved)/environ | grep PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

in 2.37.1+18.04 :

~# strings /proc/$(pidof /lib/systemd/systemd-resolved)/environ | grep PATH
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/snap/bin

[Oliver Grawert (ogra)]

likely fallout of the fix from 1771858

[E. MAS (erwan-mas)]

This is not a duplicate of 1771858 . This a critical regression from update for fixing 1771858 .

Since we have the bug we can not deploy anymoore kubernetes charm fro canonical .

[E. MAS (erwan-mas)]

This is related to 1771858 , i agree .

[Alexander Turek (turekaj)]

agree, cannot deploy Kubernetes charm from canonical, or add Kubernetes-worker units to existing deployment

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitri John Ledkov (xnox)]

Hi,

I cannot reproduce the issue, but it might be relevant how the system is booted and/or was upgraded.

Can you do the following, please?

sudo systemd-run --unit=check-env /usr/bin/env
sudo journalctl --no-pager -u check-env
cat /etc/systemd/system.conf
cat /proc/cmdline
journalctl -b -k | grep -e initrd -e command
ls /usr/lib/systemd/system-environment-generators/

And paste the output here?

[Michael Vogt (mvo)]

Here is what I see in a clean VM:

ubuntu@autopkgtest:~$ apt list snapd
Listing... Done
snapd/bionic-updates 2.33.1+18.04ubuntu2 amd64
N: There is 1 additional version. Please use the '-a' switch to see it

ubuntu@autopkgtest:~$ sudo systemd-run --pty --wait '/bin/echo' '$PATH'
Running as unit: run-u7.service
Press ^] three times within 1s to disconnect TTY.
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 13ms

After just upgrading snapd (latest version) and rebooting:

ubuntu@autopkgtest:~$ apt list snapd
Listing... Done
snapd/bionic-updates,now 2.37.1+18.04 amd64 [installed]

ubuntu@autopkgtest:~$ sudo systemd-run --pty --wait '/bin/echo' '$PATH'
Running as unit: run-u7.service
Press ^] three times within 1s to disconnect TTY.
/sbin:/usr/sbin:/bin:/usr/bin:/snap/bin
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 15ms

So this appears to be real. We added the systemd environment generator (/usr/lib/systemd/system-environment-generators/snapd-env-generator) in 2.35.1 so that seems like a likely culprit. We also ship this in 18.10 where it does not have these ill effects (but there is also a different systemd version used there).

[Michael Vogt (mvo)]

As for the questions of xnox:
"""
ubuntu@autopkgtest:~$ sudo systemd-run --unit=check-env /usr/bin/env
Running as unit: check-env.service
ubuntu@autopkgtest:~$ sudo journalctl --no-pager -u check-env
-- Logs begin at Mon 2018-07-23 18:30:47 CEST, end at Sun 2019-02-03 13:25:21 CET. --
Feb 03 13:25:21 autopkgtest systemd[1]: Started /usr/bin/env.
Feb 03 13:25:21 autopkgtest env[543]: LANG=en_US.UTF-8
Feb 03 13:25:21 autopkgtest env[543]: PATH=/sbin:/usr/sbin:/bin:/usr/bin:/snap/bin
Feb 03 13:25:21 autopkgtest env[543]: INVOCATION_ID=4c9b05e577224d6284cc641778807a16
Feb 03 13:25:21 autopkgtest env[543]: JOURNAL_STREAM=9:18437
ubuntu@autopkgtest:~$ cat /etc/systemd/system.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See systemd-system.conf(5) for details.

[Manager]
#LogLevel=info
#LogTarget=journal-or-kmsg
#LogColor=yes
#LogLocation=no
#DumpCore=yes
#ShowStatus=yes
#CrashChangeVT=no
#CrashShell=no
#CrashReboot=no
#CtrlAltDelBurstAction=reboot-force
#CPUAffinity=1 2
#JoinControllers=cpu,cpuacct net_cls,net_prio
#RuntimeWatchdogSec=0
#ShutdownWatchdogSec=10min
#CapabilityBoundingSet=
#SystemCallArchitectures=
#TimerSlackNSec=
#DefaultTimerAccuracySec=1min
#DefaultStandardOutput=journal
#DefaultStandardError=inherit
#DefaultTimeoutStartSec=90s
#DefaultTimeoutStopSec=90s
#DefaultRestartSec=100ms
#DefaultStartLimitIntervalSec=10s
#DefaultStartLimitBurst=5
#DefaultEnvironment=
#DefaultCPUAccounting=no
#DefaultIOAccounting=no
#DefaultIPAccounting=no
#DefaultBlockIOAccounting=no
#DefaultMemoryAccounting=no
#DefaultTasksAccounting=yes
#DefaultTasksMax=
#DefaultLimitCPU=
#DefaultLimitFSIZE=
#DefaultLimitDATA=
#DefaultLimitSTACK=
#DefaultLimitCORE=
#DefaultLimitRSS=
#DefaultLimitNOFILE=
#DefaultLimitAS=
#DefaultLimitNPROC=
#DefaultLimitMEMLOCK=
#DefaultLimitLOCKS=
#DefaultLimitSIGPENDING=
#DefaultLimitMSGQUEUE=
#DefaultLimitNICE=
#DefaultLimitRTPRIO=
#DefaultLimitRTTIME=
#IPAddressAllow=
#IPAddressDeny=
ubuntu@autopkgtest:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-45-generic root=UUID=991dab44-ecab-46c0-be75-150dbea660be ro console=ttyS0
ubuntu@autopkgtest:~$ journalctl -b -k | grep -e initrd -e command
Feb 03 13:23:52 autopkgtest kernel: Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-45-generic root=UUID=991dab44-ecab-46c0-be75-150dbea660be ro console=ttyS0
Feb 03 13:23:52 autopkgtest kernel: Freeing initrd memory: 50848K
ubuntu@autopkgtest:~$ ls /usr/lib/systemd/system-environment-generators/
snapd-env-generator
"""

[Łukasz Zemczak (sil2100)]

Hello E., or anyone else affected,

Accepted snapd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snapd/2.37.1.1+18.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Will Cooke (willcooke)]

Performing the same tests as mvo. Before upgrading...

$ apt list snapd
Listing... Done
snapd/bionic-updates 2.37.1+18.04 amd64 [upgradable from: 2.35.5+18.04]
N: There are 2 additional versions. Please use the '-a' switch to see them.

$ sudo systemd-run --pty --wait '/bin/echo' '$PATH'
Running as unit: run-u920.service
Press ^] three times within 1s to disconnect TTY.
/sbin:/usr/sbin:/bin:/usr/bin:/snap/bin
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 10ms

[Will Cooke (willcooke)]

After upgrade:

$ apt list snapd
Listing... Done
snapd/bionic-updates,now 2.37.1+18.04 amd64 [installed]
N: There is 1 additional version. Please use the '-a' switch to see it

$ sudo systemd-run --pty --wait '/bin/echo' '$PATH'
[sudo] password for will:
Running as unit: run-u121.service
Press ^] three times within 1s to disconnect TTY.
/sbin:/usr/sbin:/bin:/usr/bin:/snap/bin
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 8ms

[Will Cooke (willcooke)]

Ignore the previous comment, I still had the wrong version installed. Let's try that again...

$ apt list snapd
Listing... Done
snapd/bionic-proposed,now 2.37.1.1+18.04 amd64 [installed]
N: There are 2 additional versions. Please use the '-a' switch to see them.

$ sudo systemd-run --pty --wait '/bin/echo' '$PATH'
[sudo] password for will:
Running as unit: run-u120.service
Press ^] three times within 1s to disconnect TTY.
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 13ms

[Will Cooke (willcooke)]

The path is correct again. Marking as verification-done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package snapd - 2.37.1.1+18.04

---------------
snapd (2.37.1.1+18.04) bionic; urgency=medium

  * New upstream release, LP: #1811233
    - disable systemd environment generator on bionic to fix
      LP: #1814355

 -- Michael Vogt <email address hidden> Sun, 03 Feb 2019 15:20:57 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for snapd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[E. MAS (erwan-mas)]

The release of 2.37.1.1 fixed my issue .

[Dimitri John Ledkov (xnox)]

Also initramfs-tools sets PATH without /usr/local, which may leak into init environment. Fixing this as well.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.131ubuntu17

---------------
initramfs-tools (0.131ubuntu17) disco; urgency=medium

  * Fix PATH to include /usr/local. LP: #1814355

 -- Dimitri John Ledkov <email address hidden> Tue, 12 Feb 2019 13:16:06 +0000