PATH broken in systemd units

[Scott Moser]

systemd units have broken PATH in recent cloud images.
The PATH set for them is
  PATH=:/snap/bin.
Previously value is
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# cat /etc/cloud/build.info
build_name: server
serial: 20180906

Booted in lxc that fails like:
# lxc launch ubuntu-daily:cosmic c3
# lxc exec c3 cat /run/cloud-init/result.json
{
 "v1": {
  "datasource": null,
  "errors": [
   "Unexpected error while running command.\nCommand: ['netplan', 'generate']\nExit code: -\nReason: [Errno 2] No such file or directory: b'netplan': b'netplan'\nStdout: -\nStderr: -",
   "Unexpected error while running command.\nCommand: ['netplan', 'generate']\nExit code: -\nReason: [Errno 2] No such file or directory: b'netplan': b'netplan'\nStdout: -\nStderr: -",
   "('ssh-authkey-fingerprints', KeyError('getpwnam(): name not found: ubuntu',))"
  ]
 }
}

Related bugs:
 * bug 1771382: ds-identify: does not have expected PATH.

[Scott Moser (smoser)]

This regressed between 5ee1af40be67 (serial: 20180905) and 20180906.

[Philip Roche (philroche)]

I can replicate

[Dan Watkins (oddbloke)]

http://cloud-images.ubuntu.com/cosmic/20180906/unpacked/cosmic-daily.changelog outlines the changed packages between the two images. I strongly suspect that the root cause of this is the systemd environment generator added in the latest version of snapd, so I've added snapd to this bug.

The changelog for the latest snapd version:

snapd (2.35.1+18.10) cosmic; urgency=medium

  * New upstream release, LP: #1786438
    - packaging/fedora: Merge changes from Fedora Dist-Git
    - snapcraft: do not use --diry in mkversion.sh
    - cmd: add systemd environment generator
    - snap-confine: map /var/lib/extrausers into snaps mount-namespace
    - tests: cherry-pick test fixes from master for 2.35
    - systemd: do not run "snapd.snap-repair.service.in on firstboot
      bootstrap
    - interfaces: retain order of inserted security backends
    - selftest: detect if apparmor is unusable and error

 -- Michael Vogt <email address hidden> Mon, 03 Sep 2018 14:44:06 +0200

[Dimitri John Ledkov (xnox)]

#!/bin/sh
if [ -z "$PATH" ]
then
 echo PATH=/snap/bin
else
 echo PATH=$PATH:/snap/bin
fi

is the more correct minimal implementation. Note that for the case of no-path set (initrmafs-less boot, lxd boot) the PATH environment variable is not set yet; but can be prepended or expanded.

E.g. echo PATH=\$PATH:/snap/bin; is also a valid case for empty $PATH

however, output of PATH=:/snap/bin, is not. as that clears PATH, does not cause PATH expansion and injects PWD into PATH which is a security vulnerability typically.

Test case:

   # PATH= /usr/lib/systemd/system-environment-generators/snapd-env-generator
   PATH=:/snap/bin

... is bad

   # PATH= /usr/lib/systemd/system-environment-generators/snapd-env-generator
   PATH=/snap/bin

... is good

[Dimitri John Ledkov (xnox)]

This is a security issue too.

[Scott Moser (smoser)]

https://github.com/snapcore/snapd/pull/5808

[Scott Moser (smoser)]

Summary: environment-generators inherit environment that /sbin/init
started with, but no other systemd defaults. Therefore, there is no
easy way to append to an environment variable that is not set as snapd
was hoping to do. The only solution currently is to copy the expected
"default path" into the snapd environment-generator.

Long winded comment following.

I've attached an example environment-generator above, and then with it
installed and a 'print-path.service', we can see that the proposed solution in
https://github.com/snapcore/snapd/pull/5808 does not work.

To quote xnox from https://bugs.launchpad.net/cloud-init/+bug/1771382
  "Systemd by default executes things, with execv, not execve. Hence the default environment is not available."

Here is some output collected from the 4 MODE values in a container, followed by in a kvm guest booted with an initramfs.

In both cases there are no other environment generators enabled.

The inherited PATH is as follows:
  initramfs boot: PATH=/sbin:/usr/sbin:/bin:/usr/bin:/snapd/bin
  lxd container: PATH=None # not present

----- container -----

$ lxc exec c3 -- cat /run/path-generator-test.log
current PATH=None
mode=append-ref
writing 'PATH=$PATH:/snapd/bin'
=== printing ===
found PATH=$PATH:/snapd/bin

$ lxc exec c3 -- cat /run/path-generator-test.log
current PATH=None
mode=append-full
writing 'PATH=:/snapd/bin'
=== printing ===
found PATH=:/snapd/bin

$ lxc exec c3 -- cat /run/path-generator-test.log
current PATH=None
mode=none
writing ''
=== printing ===
found PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

$ lxc exec c3 -- cat /run/path-generator-test.log
current PATH=None
mode=snapd-only
writing 'PATH=/snap/bin'
=== printing ===
found PATH=/snap/bin

------ kvm guest -----
$ cat /run/path-generator-test.log
current PATH=/sbin:/usr/sbin:/bin:/usr/bin
mode=append-ref
writing 'PATH=$PATH:/snapd/bin'
=== printing ===
found PATH=$PATH:/snapd/bin

$ cat /run/path-generator-test.log
current PATH=/sbin:/usr/sbin:/bin:/usr/bin
mode=append-full
writing 'PATH=/sbin:/usr/sbin:/bin:/usr/bin:/snapd/bin'
=== printing ===
found PATH=/sbin:/usr/sbin:/bin:/usr/bin:/snapd/bin

$ cat /run/path-generator-test.log
current PATH=/sbin:/usr/sbin:/bin:/usr/bin
mode=snapd-only
writing 'PATH=/snap/bin'
=== printing ===
found PATH=/snap/bin

$ cat /run/path-generator-test.log
current PATH=/sbin:/usr/sbin:/bin:/usr/bin
mode=snapd-only
writing 'PATH=/snap/bin'
=== printing ===
found PATH=/snap/bin

[Scott Moser (smoser)]

This is reported fixed under bug 1786438 in snapd 2.35.2+18.10 which is currently in cosmic-proposed.

snapd (2.35.2+18.10) cosmic; urgency=medium

  * New upstream release, LP: #1786438
    - cmd,overlord/snapstate: go 1.11 format fixes
    - ifacestate: fix hang when retrying content providers
    - snap-env-generator: do nothing when PATH is unset
    - interfaces/modem-manager: allow access to more USB strings

 -- Michael Vogt <email address hidden> Wed, 12 Sep 2018 09:32:00 +0200

[Michael Vogt (mvo)]

Removing block-promote tag as this is fixed in 2.35.2+18.10 now (well, "fixed" it will do nothing if there is no PATH).