PATH broken in systemd units

Bug #1791691 reported by Scott Moser on 2018-09-10
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
cloud-images
Critical
Unassigned
initramfs-tools (Ubuntu)
Undecided
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Undecided
Unassigned
snapd (Ubuntu)
Critical
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Critical
Unassigned

Bug Description

systemd units have broken PATH in recent cloud images.
The PATH set for them is
  PATH=:/snap/bin.
Previously value is
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# cat /etc/cloud/build.info
build_name: server
serial: 20180906

Booted in lxc that fails like:
# lxc launch ubuntu-daily:cosmic c3
# lxc exec c3 cat /run/cloud-init/result.json
{
 "v1": {
  "datasource": null,
  "errors": [
   "Unexpected error while running command.\nCommand: ['netplan', 'generate']\nExit code: -\nReason: [Errno 2] No such file or directory: b'netplan': b'netplan'\nStdout: -\nStderr: -",
   "Unexpected error while running command.\nCommand: ['netplan', 'generate']\nExit code: -\nReason: [Errno 2] No such file or directory: b'netplan': b'netplan'\nStdout: -\nStderr: -",
   "('ssh-authkey-fingerprints', KeyError('getpwnam(): name not found: ubuntu',))"
  ]
 }
}

Related bugs:
 * bug 1771382: ds-identify: does not have expected PATH.

Related branches

Scott Moser (smoser) on 2018-09-10
Changed in cloud-images:
status: New → Confirmed
importance: Undecided → Critical
description: updated
Scott Moser (smoser) wrote :

This regressed between 5ee1af40be67 (serial: 20180905) and 20180906.

Philip Roche (philroche) wrote :

I can replicate

Dan Watkins (daniel-thewatkins) wrote :

http://cloud-images.ubuntu.com/cosmic/20180906/unpacked/cosmic-daily.changelog outlines the changed packages between the two images. I strongly suspect that the root cause of this is the systemd environment generator added in the latest version of snapd, so I've added snapd to this bug.

The changelog for the latest snapd version:

snapd (2.35.1+18.10) cosmic; urgency=medium

  * New upstream release, LP: #1786438
    - packaging/fedora: Merge changes from Fedora Dist-Git
    - snapcraft: do not use --diry in mkversion.sh
    - cmd: add systemd environment generator
    - snap-confine: map /var/lib/extrausers into snaps mount-namespace
    - tests: cherry-pick test fixes from master for 2.35
    - systemd: do not run "snapd.snap-repair.service.in on firstboot
      bootstrap
    - interfaces: retain order of inserted security backends
    - selftest: detect if apparmor is unusable and error

 -- Michael Vogt <email address hidden> Mon, 03 Sep 2018 14:44:06 +0200

Changed in snapd (Ubuntu):
importance: Undecided → Critical
Dimitri John Ledkov (xnox) wrote :

#!/bin/sh
if [ -z "$PATH" ]
then
 echo PATH=/snap/bin
else
 echo PATH=$PATH:/snap/bin
fi

is the more correct minimal implementation. Note that for the case of no-path set (initrmafs-less boot, lxd boot) the PATH environment variable is not set yet; but can be prepended or expanded.

E.g. echo PATH=\$PATH:/snap/bin; is also a valid case for empty $PATH

however, output of PATH=:/snap/bin, is not. as that clears PATH, does not cause PATH expansion and injects PWD into PATH which is a security vulnerability typically.

Test case:

   # PATH= /usr/lib/systemd/system-environment-generators/snapd-env-generator
   PATH=:/snap/bin

... is bad

   # PATH= /usr/lib/systemd/system-environment-generators/snapd-env-generator
   PATH=/snap/bin

... is good

tags: added: block-proposed
Dimitri John Ledkov (xnox) wrote :

This is a security issue too.

information type: Public → Public Security
Michael Vogt (mvo) on 2018-09-10
Changed in snapd (Ubuntu Bionic):
status: New → In Progress
Changed in snapd (Ubuntu Cosmic):
status: New → In Progress
Scott Moser (smoser) wrote :
Scott Moser (smoser) wrote :

Summary: environment-generators inherit environment that /sbin/init
started with, but no other systemd defaults. Therefore, there is no
easy way to append to an environment variable that is not set as snapd
was hoping to do. The only solution currently is to copy the expected
"default path" into the snapd environment-generator.

Long winded comment following.

I've attached an example environment-generator above, and then with it
installed and a 'print-path.service', we can see that the proposed solution in
https://github.com/snapcore/snapd/pull/5808 does not work.

To quote xnox from https://bugs.launchpad.net/cloud-init/+bug/1771382
  "Systemd by default executes things, with execv, not execve. Hence the default environment is not available."

Here is some output collected from the 4 MODE values in a container, followed by in a kvm guest booted with an initramfs.

In both cases there are no other environment generators enabled.

The inherited PATH is as follows:
  initramfs boot: PATH=/sbin:/usr/sbin:/bin:/usr/bin:/snapd/bin
  lxd container: PATH=None # not present

----- container -----

$ lxc exec c3 -- cat /run/path-generator-test.log
current PATH=None
mode=append-ref
writing 'PATH=$PATH:/snapd/bin'
=== printing ===
found PATH=$PATH:/snapd/bin

$ lxc exec c3 -- cat /run/path-generator-test.log
current PATH=None
mode=append-full
writing 'PATH=:/snapd/bin'
=== printing ===
found PATH=:/snapd/bin

$ lxc exec c3 -- cat /run/path-generator-test.log
current PATH=None
mode=none
writing ''
=== printing ===
found PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

$ lxc exec c3 -- cat /run/path-generator-test.log
current PATH=None
mode=snapd-only
writing 'PATH=/snap/bin'
=== printing ===
found PATH=/snap/bin

------ kvm guest -----
$ cat /run/path-generator-test.log
current PATH=/sbin:/usr/sbin:/bin:/usr/bin
mode=append-ref
writing 'PATH=$PATH:/snapd/bin'
=== printing ===
found PATH=$PATH:/snapd/bin

$ cat /run/path-generator-test.log
current PATH=/sbin:/usr/sbin:/bin:/usr/bin
mode=append-full
writing 'PATH=/sbin:/usr/sbin:/bin:/usr/bin:/snapd/bin'
=== printing ===
found PATH=/sbin:/usr/sbin:/bin:/usr/bin:/snapd/bin

$ cat /run/path-generator-test.log
current PATH=/sbin:/usr/sbin:/bin:/usr/bin
mode=snapd-only
writing 'PATH=/snap/bin'
=== printing ===
found PATH=/snap/bin

$ cat /run/path-generator-test.log
current PATH=/sbin:/usr/sbin:/bin:/usr/bin
mode=snapd-only
writing 'PATH=/snap/bin'
=== printing ===
found PATH=/snap/bin

description: updated
Scott Moser (smoser) wrote :

This is reported fixed under bug 1786438 in snapd 2.35.2+18.10 which is currently in cosmic-proposed.

snapd (2.35.2+18.10) cosmic; urgency=medium

  * New upstream release, LP: #1786438
    - cmd,overlord/snapstate: go 1.11 format fixes
    - ifacestate: fix hang when retrying content providers
    - snap-env-generator: do nothing when PATH is unset
    - interfaces/modem-manager: allow access to more USB strings

 -- Michael Vogt <email address hidden> Wed, 12 Sep 2018 09:32:00 +0200

Michael Vogt (mvo) wrote :

Removing block-promote tag as this is fixed in 2.35.2+18.10 now (well, "fixed" it will do nothing if there is no PATH).

tags: removed: block-proposed
Michael Vogt (mvo) on 2018-09-24
Changed in snapd (Ubuntu Cosmic):
status: In Progress → Fix Released
Changed in cloud-images:
status: Confirmed → In Progress
status: In Progress → Incomplete
status: Incomplete → Fix Committed
status: Fix Committed → New
status: New → Fix Released
affects: cloud-images → arduino-debug
Colin Watson (cjwatson) on 2019-04-02
affects: arduino-debug → cloud-images
Changed in cloud-images:
status: Fix Released → Confirmed
information type: Public Security → Public
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers