16.04.3 deployments include non-existent directories in $PATH

[Jeff Lane ]

Noticed this while doing regression testing on 16.04.3, and a number of customers have asked with concerns as well during their testing.

I don't know exactly when this started, but we just started noticing it with 16.04.3 (4.10) but $PATH by default now includes several directories that do not exist:

ubuntu@above-ox:~$ find $(echo "$PATH" | sed -e 's/:/ /g') -maxdepth 1 -type d
find: ‘/home/ubuntu/bin’: No such file or directory
find: ‘/home/ubuntu/.local/bin’: No such file or directory
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/usr/games
/usr/local/games
find: ‘/snap/bin’: No such file or directory

We noticed this because it was causing a resource job in the certification suite to suddenly start failing (the failure was not catastrophic, but it did appear to customers who were concerned.

I've verified this on both s390x and amd64 16.04.3 deployments via MAAS 2.2.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.10.0-32-generic 4.10.0-32.36~16.04.1
ProcVersionSignature: Ubuntu 4.10.0-32.36~16.04.1-generic 4.10.17
Uname: Linux 4.10.0-32-generic s390x
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: s390x
Date: Wed Aug 16 18:59:34 2017
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en.US_UTF-8
 SHELL=/bin/bash
SourcePackage: linux-hwe
UpgradeStatus: No upgrade log present (probably fresh install)

[Dimitri John Ledkov (xnox)]

~/bin ~/.local/bin used to be added in a user path only if they exist.
/snap/bin may or or may not exist, but could be a bug in snapd.

[Adam Conrad (adconrad)]

The user bin bits of this are due to https://bugs.launchpad.net/ubuntu/+source/bash/+bug/1588562

We can have it one way or the other, but not both ways. Not sure I agree with having nonexistent paths in PATH, but it's shouldn't technically break anything either.

[Steve Langasek (vorlon)]

~/.bin and ~/.local/bin are added to the path by /etc/skel/.profile which is shipped in the bash package.

It's debatable whether these should be added to the path unconditionally, or only if they exist. It is convenient to have the directories automatically available on the path as soon as they are created, without having to relogin / re-source the shell config; conversely, it means a couple of extra pointless stats on every exec() call.

Regardless, though, I don't think it's correct that your certification suite should throw errors as a result of directories on the PATH not being present on the filesystem.

[Jeff Lane  (bladernr)]

@Steve, I don't disagree, and I've already merged a patch to fix that issue in the cert suite.

For context, what happens isn't a test case, but a resource that enumerates the binaries in the system so that the suite can validate requirements for executing the tests.

e.g., it enumerates a list that contains "/usr/bin/iperf3" for example, and the network test then has a constraint to only execute if "/usr/bin/iperf3" actually exists.

The failure was that the enumeration just executes a 'find', the example noted in the summary is almost exactly the command the resource job uses, and find exits with a fail code because the directories listed in $PATH didn't exist. I couldn't find a way to tell find to simply ignore non-existent things, without turning a shell oneliner into a more complex script, so in the end, the resource job now always returns "true" regardless of the exit code from 'find'.

But since customers started voicing concern about it, and because it just seemed odd to me for us to be automatically injecting false data into $PATH (for whatever reason, I read the bug Adam pointed out, and I can understand now why this happened) I thought it was worth raising a bug, even if the ultimate answer is "there were reasons for doing it that couldn't be avoided".

[gdahlman (gdahlman)]

Note in reference to ~/.local/bin this path is added because it is a systemd standard directory.

They are documented in file-hierarchy(7) and also here.

https://www.freedesktop.org/software/systemd/man/file-hierarchy.html

If the issue is that the paths don't exist they should be created for the user as methods that leverage systemd-path will fail.

$ systemd-path user-binaries

It is also the bin directory for official Ubuntu packages like virtualenv.