ssh can't call a binary from a snap without the full path

Bug #1659719 reported by Leo Arias on 2017-01-27
54
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Snappy
Medium
Oliver Grawert
livecd-rootfs (Ubuntu)
Medium
Oliver Grawert
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Focal
Undecided
Unassigned
Groovy
Medium
Oliver Grawert
openssh (Debian)
New
Unknown
openssh (Ubuntu)
Undecided
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Focal
Undecided
Unassigned
Groovy
Undecided
Unassigned
pam (Ubuntu)
Medium
Unassigned
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Focal
Undecided
Unassigned
Groovy
Medium
Unassigned

Bug Description

[impact]
ssh can't call a binary from a snap, it will only work using the full path.

[test case]
Create a container. Install the go snap (and make sure golang-go is not installed). Run "ssh <container-ip> go version" and check the binary is found.

[regression potential]
It's a pam change an they are always a bit scary but the code follows the existing pattern for updating PATH in /etc/environment and has been tested in groovy.

[original description]

Let's say I have the hello snap installed in 192.168.122.24. Then:

elopio@ubuntu-xenial:~/mosh$ ssh 192.168.122.24 hello
elopio@192.168.122.24's password:
bash: hello: command not found
elopio@ubuntu-xenial:~/mosh$ ssh 192.168.122.24 /snap/bin/hello
elopio@192.168.122.24's password:
Hello, world!

Leo Arias (elopio) wrote :
tags: added: isv
Leo Arias (elopio) wrote :

Comment from ogra in the mailing list:

bash actually checks the calling process and will not source the env if
it is invoked by sshd or rshd as non-login shell (it checks if
the SSH_CLIENT environment variable is set and uses a different code
path internally).

the follwing works:

$ ssh 192.168.2.91 env|grep PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/
games:/usr/local/games

$ ssh 192.168.2.91 bash -lc env|grep PATH
PATH=/home/ogra/bin:/home/ogra/.local/bin:/usr/local/sbin:/usr/local/bi
n:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

you could also do something like:
ssh 192.168.2.91 "source /etc/profile; hello"

i dont think we have a bug open for this and technically it is expected
behaviour (not different from any other ubuntu install), but given how
annoying it is i guess you should file one and we should find a proper
workaround or fix to ship in the image ...

Seth Arnold (seth-arnold) wrote :

Why is the PATH being set via login shells rather than e.g. init's environment variables or PAM environment mechanism? What led to setting the environment variables via shell startup scripts?

Thanks

Dan Kegel (dank) wrote :

 /etc/profile.d/apps-bin-path.sh does not fire on noninteractive logins.

$ ssh pi3 'echo $PATH'
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

Workaround: make it fire, dammit:

--- .bashrc.orig 2017-01-04 19:55:33.105726685 +0000
+++ .bashrc 2017-01-04 19:54:11.505653043 +0000
@@ -5,7 +5,9 @@
 # If not running interactively, don't do anything
 case $- in
     *i*) ;;
- *) return;;
+ *) # Work around rsync snap PATH problem
+ . /etc/profile.d/apps-bin-path.sh
+ return;;
 esac

 # don't put duplicate lines or lines starting with space in the history.

Oliver Grawert (ogra) wrote :

@seth because it is dynamically shipped with the snapd package for classic installs so PATH gets automatically extended with install/removal of the snapd package.

for image builds i am now adding it to /etc/environment (which is traditionally the place for the global PATH variable in debian and Ubuntu) at build time, that should fix the issue for the images at least.

Oliver Grawert (ogra) on 2017-01-27
Changed in snappy:
status: New → Confirmed
importance: Undecided → Medium
assignee: nobody → Oliver Grawert (ogra)
Changed in livecd-rootfs (Ubuntu):
status: New → Fix Committed
Changed in snappy:
status: Confirmed → Fix Committed
Changed in livecd-rootfs (Ubuntu):
importance: Undecided → Medium
assignee: nobody → Oliver Grawert (ogra)
Oliver Grawert (ogra) wrote :

so with todays edge core snap it now looks like:

$ ssh 192.168.2.76 env |grep PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
$

Greg Lutostanski (lutostag) wrote :

Still hitting this with...

jenkins@juju-6f7422-solutions-qa-integration-1:~$ snap version
snap 2.27.6
snapd 2.27.6
series 16
ubuntu 16.04
kernel 4.4.0-83-generic
jenkins@juju-6f7422-solutions-qa-integration-1:~$ snap list
Name Version Rev Developer Notes
conjure-up 2.3.1 741 canonical classic
core 16-2.27.6 2898 canonical core
weebl-tools 1.1.8 50 autonomouse -

was it simply never released, or did we regress somehow, or am I missing something else?

Oliver Grawert (ogra) wrote :

this has been fixed long ago and you should have an /etc/profile.d/apps-bin-path.sh file that adds /snap/bin to your PATH

Greg Lutostanski (lutostag) wrote :

Yes it obviously works for login shells.

To make it work for non-login shells /etc/environment would need to include /snap/bin -- an issue for foundations to decide upon.

Thanks for the insight.

Robie Basak (racb) wrote :

Adding a task against the Ubuntu snapd package for the non-interactive shell case (eg. ssh), since this still affects Ubuntu the distribution on this point.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in snapd (Ubuntu):
status: New → Confirmed
Colin Watson (cjwatson) wrote :

I'm very reluctant to reconfigure openssh for this; it feels like a layering violation, and a regression of all the work we did way back when to avoid having to change umpteen packages when we want to change the default command search path.

How about something like adding a pam_snap session module that would add itself to PATH in the PAM environment if needed, and plug that in via /usr/share/pam-configs/ ? You'd want to discuss that sort of thing with Steve Langasek, I expect, but it seems nicer than changing the configuration of individual PAM-using services.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openssh (Ubuntu):
status: New → Confirmed
Johan Ehnberg (johan-ehnberg) wrote :

This is fairly problematic when core functionality that is orchestrated in the cloud moves from deb to snap. In our example, lxc (LXD command line tool) was assumed to be in the path when building the orchestration. Using the full path makes it less portable. It also makes managing multiple releases a pain since some hosts may run the snap, others installed by dpkg.

Is there any update on the intended direction?

Steve Langasek (vorlon) wrote :

I know that work was done to ensure /snap/bin was in the default path for systemd itself and that this should be inherited now by all units. 'systemd-path | grep search-binaries:' shows that it's there on my focal system. 'sudo ps axewwww | grep sshd' also shows that it's set in the PATH environment of the sshd parent processes. So I'm not sure what is still resetting the PATH for noninteractive ssh sessions.

Steve Langasek (vorlon) wrote :

It looks like the likely cause here is that sshd loads pam, which calls pam_env, which reads /etc/environment, which has a different PATH than the default one provided by systemd.

We should fix this to be aligned.

Paride Legovini (paride) wrote :

+1, tweaking /etc/environment makes it work. It seems that this has to be fixed in the libpam-modules postinst script then.

Changed in pam (Ubuntu):
status: New → Confirmed
tags: added: rls-gg-incoming
Changed in livecd-rootfs (Ubuntu):
status: Fix Committed → Fix Released
tags: removed: rls-gg-incoming
Changed in pam (Ubuntu Groovy):
importance: Undecided → Medium
tags: added: id-5ec69c5a5bc4156ecefb75a0

It's easy enough to unconditionally add /snap/bin to path, see attached, but I can't really see a clean way to only have it present in the $PATH when snapd is installed (without code changes to pam_env, anyway).

tags: added: patch
Paride Legovini (paride) wrote :

That won't be too different from /usr/local/sbin and /usr/local/bin being in the PATH but empty I think. I am not aware of any problem or annoyance caused by nonexisting directories being in the PATH.

Robie Basak (racb) wrote :

Given Colin's comment 12, it doesn't look like there are any plans to change this in openssh packaging so I'll mark this Won't Fix.

Changed in openssh (Ubuntu Groovy):
status: Confirmed → Won't Fix
Steve Langasek (vorlon) wrote :

The inconsistency between login and openssh session module ordering is still an issue. Reopening.

Changed in openssh (Ubuntu Groovy):
status: Won't Fix → Confirmed

Here's a better patch that I've tested this time. I dropped the code around adding /usr/local/games to PATH as that was in trusty.

Changed in pam (Ubuntu Groovy):
status: Confirmed → In Progress

I've filed https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=964580 about the pam ordering issue.

no longer affects: snapd (Ubuntu)
no longer affects: snapd (Ubuntu Groovy)
Changed in openssh (Debian):
status: Unknown → New
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package pam - 1.3.1-5ubuntu5

---------------
pam (1.3.1-5ubuntu5) groovy; urgency=medium

  * debian/libpam-modules.postinst: Add /snap/bin to $PATH in
    /etc/environment. (LP: #1659719)

 -- Michael Hudson-Doyle <email address hidden> Fri, 10 Jul 2020 08:35:49 +1200

Changed in pam (Ubuntu Groovy):
status: In Progress → Fix Released
description: updated

Hello Leo, or anyone else affected,

Accepted pam into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pam/1.3.1-5ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in pam (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed verification-needed-focal

All autopkgtests for the newly accepted pam (1.3.1-5ubuntu4.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

kopanocore/8.7.0-7ubuntu1 (armhf)
cyrus-imapd/3.0.13-5 (armhf, arm64, s390x, ppc64el, amd64)
xpra/3.0.6+dfsg1-1build1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#pam

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hello Leo, or anyone else affected,

Accepted pam into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pam/1.1.8-3.6ubuntu2.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in pam (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed-bionic

All autopkgtests for the newly accepted pam (1.1.8-3.6ubuntu2.18.04.2) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

dovecot/1:2.2.33.2-1ubuntu4.5 (armhf)
systemd/237-3ubuntu10.41 (ppc64el)
openssh/1:7.6p1-4ubuntu0.3 (arm64, amd64, i386, armhf, s390x, ppc64el)
vsftpd/3.0.3-9build1 (arm64)
lxc/3.0.3-0ubuntu1~18.04.1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#pam

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Verified on focal:

mwhudson@anduril:~$ ssh ubuntu@10.187.142.128 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.3.1-5ubuntu4
go: not found
mwhudson@anduril:~$ lxc exec test-pam-focal -- apt install -y libpam-modules/focal-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '1.3.1-5ubuntu4.1' (Ubuntu:20.04/focal-proposed [amd64]) for 'libpam-modules'
Selected version '1.3.1-5ubuntu4.1' (Ubuntu:20.04/focal-proposed [amd64]) for 'libpam-modules-bin' because of 'libpam-modules'
...
mwhudson@anduril:~$ ssh ubuntu@10.187.142.128 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.3.1-5ubuntu4.1
go is /snap/bin/go

The focal autopkgtest failures are in cyrus-imapd:

Mail::JMAPTalk version 0.15 required--this is only version 0.13 at ./Cassandane/Cyrus/JMAPCore.pm line 47.
BEGIN failed--compilation aborted at ./Cassandane/Cyrus/JMAPCore.pm line 47.

This seems very unlikely to be due to pam, and it happens with the version of pam anyway. The other failure is kopanocore/armhf which is known flaky: https://autopkgtest.ubuntu.com/packages/k/kopanocore/focal/armhf (I've retried it anyway).

tags: added: verification-done-focal
removed: verification-needed-focal

And bionic:

mwhudson@anduril:~$ ssh ubuntu@10.187.142.63 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.1.8-3.6ubuntu2.18.04.1
go: not found
mwhudson@anduril:~$ lxc exec test-pam-bionic -- apt install -y libpam-modules/bionic-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '1.1.8-3.6ubuntu2.18.04.2' (Ubuntu:18.04/bionic-proposed [amd64]) for 'libpam-modules'
Selected version '1.1.8-3.6ubuntu2.18.04.2' (Ubuntu:18.04/bionic-proposed [amd64]) for 'libpam-modules-bin' because of 'libpam-modules'
...
mwhudson@anduril:~$ ssh ubuntu@10.187.142.63 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.1.8-3.6ubuntu2.18.04.2
go is /snap/bin/go

The autopkgtest failure, dovecot/armhf looks to be a flake. I've retried it anyway to see if it passes now.

tags: added: verification-done-bionic
removed: verification-needed verification-needed-bionic
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package pam - 1.1.8-3.6ubuntu2.18.04.2

---------------
pam (1.1.8-3.6ubuntu2.18.04.2) bionic; urgency=medium

  * debian/libpam-modules.postinst: Add /snap/bin to $PATH in
    /etc/environment. (LP: #1659719)

 -- Michael Hudson-Doyle <email address hidden> Wed, 22 Jul 2020 11:40:59 +1200

Changed in pam (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for pam has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Brian Murray (brian-murray) wrote :

If kopanocore is known to be flaky on armhf please add a hint for it to https://code.launchpad.net/~ubuntu-sru/britney/hints-ubuntu-focal/.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package pam - 1.3.1-5ubuntu4.1

---------------
pam (1.3.1-5ubuntu4.1) focal; urgency=medium

  * debian/libpam-modules.postinst: Add /snap/bin to $PATH in
    /etc/environment. (LP: #1659719)

 -- Michael Hudson-Doyle <email address hidden> Wed, 22 Jul 2020 11:30:44 +1200

Changed in pam (Ubuntu Focal):
status: Fix Committed → Fix Released
Steve Langasek (vorlon) on 2020-08-31
Changed in openssh (Ubuntu Xenial):
status: New → Won't Fix
Changed in openssh (Ubuntu Bionic):
status: New → Won't Fix
Changed in openssh (Ubuntu Focal):
status: New → Won't Fix
Brian Murray (brian-murray) wrote :

The fix for this in the queue for Xenial is help up by the verification of bug 1666203.

An upload of pam to xenial-proposed has been rejected from the upload queue for the following reason: "needs sourceful fixes for the version previously in -proposed".

Hello Leo, or anyone else affected,

Accepted pam into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pam/1.1.8-3.2ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in pam (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed verification-needed-xenial

All autopkgtests for the newly accepted pam (1.1.8-3.2ubuntu2.3) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/229-4ubuntu21.29 (amd64, i386)
kscreenlocker/5.5.5-0ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#pam

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

And verified on xenial:

(client-server-5)mwhudson@anduril:~/src/subiquity$ ssh ubuntu@10.187.142.193 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.1.8-3.2ubuntu2.1
go: not found
(client-server-5)mwhudson@anduril:~/src/subiquity$ lxc exec lp1659719-xenial -- apt install -y libpam-modules/xenial-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '1.1.8-3.2ubuntu2.3' (Ubuntu:16.04/xenial-proposed [amd64]) for 'libpam-modules'
Selected version '1.1.8-3.2ubuntu2.3' (Ubuntu:16.04/xenial-proposed [amd64]) for 'libpam-modules-bin' because of 'libpam-modules'
The following packages were automatically installed and are no longer required:
  libfreetype6 motd-news-config
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
  libpam-modules-bin
Recommended packages:
  update-motd
The following packages will be upgraded:
  libpam-modules libpam-modules-bin
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 281 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 libpam-modules-bin amd64 1.1.8-3.2ubuntu2.3 [36.9 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 libpam-modules amd64 1.1.8-3.2ubuntu2.3 [244 kB]
Fetched 281 kB in 2s (123 kB/s)
Preconfiguring packages ...
(Reading database ... 25804 files and directories currently installed.)
Preparing to unpack .../libpam-modules-bin_1.1.8-3.2ubuntu2.3_amd64.deb ...
Unpacking libpam-modules-bin (1.1.8-3.2ubuntu2.3) over (1.1.8-3.2ubuntu2.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up libpam-modules-bin (1.1.8-3.2ubuntu2.3) ...
(Reading database ... 25804 files and directories currently installed.)
Preparing to unpack .../libpam-modules_1.1.8-3.2ubuntu2.3_amd64.deb ...
Unpacking libpam-modules:amd64 (1.1.8-3.2ubuntu2.3) over (1.1.8-3.2ubuntu2.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up libpam-modules:amd64 (1.1.8-3.2ubuntu2.3) ...
(client-server-5)mwhudson@anduril:~/src/subiquity$ ssh ubuntu@10.187.142.193 "sh -c 'dpkg-query -W libpam-modules; type go '"
libpam-modules:amd64 1.1.8-3.2ubuntu2.3
go is /snap/bin/go

tags: added: verification-done-xenial
removed: verification-needed verification-needed-xenial
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package pam - 1.1.8-3.2ubuntu2.3

---------------
pam (1.1.8-3.2ubuntu2.3) xenial; urgency=medium

  * Move patch fixing LP: #1666203 from debian/patches to
    debian/patches-applied so it actually gets applied.
  * debian/libpam-modules.postinst: Add /snap/bin to $PATH in
    /etc/environment. (LP: #1659719)

pam (1.1.8-3.2ubuntu2.2) xenial; urgency=medium

  * Fix: pam_tty_audit failed in pam_open_session (LP: #1666203)

 -- Michael Hudson-Doyle <email address hidden> Thu, 01 Oct 2020 10:03:21 +1300

Changed in pam (Ubuntu Xenial):
status: Fix Committed → Fix Released
tags: added: fr-78
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.