Subprocesses of StartProgramInGuest fail when creating temporary files

Bug #1758428 reported by Tim Clark
20
This bug affects 2 people
Affects Status Importance Assigned to Milestone
open-vm-tools (Debian)
Fix Released
Unknown
open-vm-tools (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
High
Unassigned
Artful
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned

Bug Description

Using open-vm-tools/xenial,now 2:10.2.0-3ubuntu0.16.04.1~ppa6 amd64 [installed], subprocesses started by vim.vm.guest.ProcessManager fail to create temporary files.

Summary:
* When running subprocesss through open-vm-tools 2:10.2.0-3ubuntu0.16.04.1 (via vim.vm.guest.ProcessManager), subprocesses crash while trying to write temporary files.
* Subprocesses run correctly when executed directly from the shell
* Subprocesses execute correctly from open-vm-tools open-vm-tools-10.2.0-7253323, when compiled from source

Steps to reproduce:
1. Install open-vm-tools/xenial 2:10.2.0-3ubuntu0.16.04.1~ppa6 on a VMWare ESX 6.5 16.04 guest
2. Using VMOMI API, start a child process that relies on temporary files (eg. `apt-get update`)
3. Note the error code on the child process (eg. 100)
4. Run process directly from guest shell (eg. `apt-get update`)
5. Note the successful return code on the child process (1)

Logs from vmtoolsd on guest:

Mar 23 19:32:32 kcp vmtoolsd[660]: Hit:1 https://apt.dockerproject.org/repo ubuntu-xenial InRelease
Mar 23 19:32:32 kcp vmtoolsd[660]: Couldn't create tempfiles for splitting up /var/lib/apt/lists/apt.dockerproject.org_repo_dists_ubuntu-xenial_InReleaseErr:1 https://apt.dockerproject.org/repo ubuntu-xenial InRelease
Mar 23 19:32:32 kcp vmtoolsd[660]: Could not execute 'apt-key' to verify signature (is gnupg installed?)

---------------------------------

Psuedo code using pyVMOMI API to start a child process:

#!/usr/bin/python
import re
import time
from pyVim import connect
from pyVmomi import vim, vmodl

service_instance = connect.SmartConnectNoSSL(host=<vcenter_hostname>,
                                            user=<vcenter_username>,
                                            pwd=<vcenter_password>,
                                            port=443)
content = service_instance.RetrieveContent()

vm = content.searchIndex.FindByIp(datacenter=None,
                                  ip=<vm_ip_address>,
                                  vmSearch=True)

creds = vim.vm.guest.NamePasswordAuthentication(
    username="root", password=<guest_password>
)

pm = content.guestOperationsManager.processManager
ps = vim.vm.guest.ProcessManager.ProgramSpec(
   programPath="/usr/bin/apt-get",
   arguments="update"
)
res = pm.StartProgramInGuest(vm, creds, ps)
pid_exitcode = pm.ListProcessesInGuest(vm, creds,
                                       [res]).pop().exitCode
while re.match('[^0-9]+', str(pid_exitcode)):
    time.sleep(5)
    pid_exitcode = pm.ListProcessesInGuest(vm, creds,
                                           [res]).pop().exitCode
    if pid_exitcode == 0:
        print("Program %d completed with success" % res)
        break
    # Look for non-zero code to fail
    elif re.match('[1-9]+', str(pid_exitcode)):
        print("ERROR: Program apt-get completed with Failure %s" % pid_exitcode)

CVE References

Revision history for this message
Oliver Kurth (okurth-1) wrote :

Tim, what exactly do you mean with 'Subprocesses execute correctly from open-vm-tools open-vm-tools-10.2.0-7253323, when compiled from source'? Is this a rebuild of the package, or is this a build of just the open-vmware-tools source without Debian package scripts?

Revision history for this message
Oliver Kurth (okurth-1) wrote :

Also, does this work with the version of open-vm-tools that ships with 16.04? Did you try the same test with 18.04?

Revision history for this message
Oliver Kurth (okurth-1) wrote :

I was able to to reproduce a similar issue by trying to use ListFilesInGuest in /tmp. We found that the issue can be fixed by commenting out the option
PrivateTmp=yes
under [Services] in /lib/systemd/system/open-vm-tools.service and the commands
# systemctl daemon-reload
# systemctl restart open-vm-tools

Revision history for this message
Oliver Kurth (okurth-1) wrote :

Okay, so I now tried to reproduce the original issue with code that is very similar to the code in the bug description, but wasn't able to.

@Tim, can you please check if unsetting PrivateTmp option fixes your issue?

Revision history for this message
Tim Clark (timothygclark) wrote :

Oliver, `PrivateTmp=yes` sounds like a good find. Will give that a try in the morning and report back.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in open-vm-tools (Ubuntu):
status: New → Confirmed
Revision history for this message
Tim Clark (timothygclark) wrote :

Still working to confirm - As restarting the service clears up the symptoms, I'm building a new base image with `PrivateTmp=false` in the service unit file.

Revision history for this message
Tim Clark (timothygclark) wrote :

@Oliver, to speak to your original question above, "Yes, our in-house compilation is a 'build of just the open-vmware-tools source without Debian package scripts'" That build doesn't appear to have that PrivateTmp in the unit file.

So far, testing looks good with 'PrivateTmp=False'. I'm going to run one more test through for verification before confirming this resolves the issue.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I'll take care that the proposed cahnge for the Xenial SRU will drop this systemd entry.
And respon the SRU upload.

But I wonder if this is an issue in Bionic as well.
The change to use PrivateTmp for open-vm-tools.service was introduced in Debian in 2:10.1.5-5055683-5 at 25 Jul 2017.

We didn't hear about this issue yet, so the question is
a) was it missed and also affects Bionic 18.04?
b) is it not an issue due to e.g. the newer systemd version?

I beg your pardon that "Using VMOMI API" is too unclear for me yet, but I assume that should be a rather simple test for you if this affects 18.04 as well.

Changed in open-vm-tools (Ubuntu Xenial):
status: New → Triaged
Changed in open-vm-tools (Ubuntu Bionic):
status: Confirmed → Incomplete
Changed in open-vm-tools (Ubuntu Xenial):
importance: Undecided → High
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

The SRU in the queue for Xenial (where this was found an tested) has the requested change.

It would now be nice to know if you could confirm (or falsify) this being an issue for Bionic as well so that we can remove it there.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I'd really need to know if Bionic is affected as well, but can't test it myself.

If it is we need/want to fix it in Bionic first (SRU rules and new LTS should be good)
If it is not in Bionic then the SRU for 1741390 is unblocked (of this issue).

Revision history for this message
Tim Clark (timothygclark) wrote :

We are still trying to confirm the fix for Xenial - our initial test appeared to work, but we've had nothing but subsequent (potentially unrelated failures). Still working to validate.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI: PPA build with the fix for privateTmp is available in a PPA
Versioning changes slightly (better to do these SRUs regularly).
So the new version is below the old one, be sure to update correctly when testing this.

=> https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3226

Still confirmation in Bionic is the most important next step I'd ask for.

Revision history for this message
David Britton (dpb) wrote :

@Tim, @Oliver -- We are quickly approaching the Release of 18.04, which will add steps and time to this SRU if we don't get it in quickly. Could one of you please test that Bionic (18.04) works with the PPA that Christian attached?

Thanks!

Revision history for this message
Tim Clark (timothygclark) wrote :

I have not had the time to return to this bug - as of last tests, `PrivateTmp=yes` did not fix the issue after all. Restarting the daemon always fixes the issue, but we're unsure why.

Revision history for this message
vmware-gos-Yuhua (yhzou) wrote :

> #ChristianEhrhardt (paelzer) wrote on 2018-04-04:
> I'd really need to know if Bionic is affected as well, but can't test it myself.

The ubuntu 18.04 with default open-vm-tools 2:10.2.0-3ubuntu2 also have this issue.
Commenting out the option PrivateTmp=yes in /lib/systemd/system/open-vm-tools.service can fix the issue.

> #David Britton (davidpbritton) wrote on 2018-04-10:
> Could one of you please test that Bionic (18.04) works with the PPA that Christian attached?

Adding ppa with command "sudo add-apt-repository ppa:ci-train-ppa-service/3226" is blocked:

E: The repository 'https://launchpad.net/ci-train-ppa-service/3226 bionic Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.

Revision history for this message
vmware-gos-Yuhua (yhzou) wrote :

using open-vm-tools in ppa:ci-train-ppa-service/3226 for ubuntu 16.04, this issue disappear.

After install open-vm-tools from repository ppa:ci-train-ppa-service/3226 and check /lib/systemd/system/open-vm-tool.service, I find "PrivateTmp=yes" is removed from file.
If I add "PrivateTmp=yes" to file open-vm-tool.service manually, the issue can be reproducible. So my guessing is that the change to fix the issue in this PPA is removing configuration item "PrivateTmp=yes" from open-vm-tool.service

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thank you for checking.
The ppa was only meant for 16.04 and yes since we knew the issue was there it has the fix.
Overall this confirms all I needed, lets make the change for Bionic before it releases and from thre make it part of the SRUs for Artful and Xenial.

Changed in open-vm-tools (Ubuntu Bionic):
status: Incomplete → In Progress
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Uploaded to Bionic, but waits in -unapproved there atm as we are in pre-release freeze.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package open-vm-tools - 2:10.2.0-3ubuntu3

---------------
open-vm-tools (2:10.2.0-3ubuntu3) bionic; urgency=medium

  * Disable PrivateTmp for the open-vm-tools.service as it triggers issues
    when triggering processes that need tmp through VMOMI API (LP: #1758428)

 -- Christian Ehrhardt <email address hidden> Thu, 19 Apr 2018 11:23:58 +0200

Changed in open-vm-tools (Ubuntu Bionic):
status: In Progress → Fix Released
Changed in open-vm-tools (Ubuntu Artful):
status: New → Triaged
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

This will only "become" an issue for Xenial/Artful with the backport.
But lets do it right for tracking - so I added tasks for these releases which allows me to refer changes and changelog to here.

That way with the backport it will "be an issue" for the former releases, but also instantly be closed for them.

In the queue for SRU review now - main bug 1741390

Revision history for this message
vmware-gos-Yuhua (yhzou) wrote :

   > Launchpad Janitor (janitor) wrote 5 hours ago:
   > This bug was fixed in the package open-vm-tools - 2:10.2.0-3ubuntu3

verified.
1) we can update to package 10.2.0-3ubuntu3 when type "sudo apt-get install open-vm-tools" in terminal. After updating, the configuration item "PrivateTmp=yes" is removing from file open-vm-tool.service

2) This issue disappear with package open-vm-tools - 2:10.2.0-3ubuntu3.

3) it works well when check basic tools functions in 18.04 Server/Desktop: install, unintalll,
   tools status, vmware-toolbox-cmd, VGauth.

Revision history for this message
Chris Halse Rogers (raof) wrote : Please test proposed package

Hello Tim, or anyone else affected,

Accepted open-vm-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-vm-tools/2:10.2.0-3~ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in open-vm-tools (Ubuntu Xenial):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Chris Halse Rogers (raof) wrote :

Hello Tim, or anyone else affected,

Accepted open-vm-tools into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-vm-tools/2:10.2.0-3~ubuntu0.17.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in open-vm-tools (Ubuntu Artful):
status: Triaged → Fix Committed
tags: added: verification-needed-artful
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

1. Upgrade from proposed - this is the same for all associated bugs, so I only documented details in bug 1741390

2. This bug in particular
While triggering the raw error throu VMOMI API is complex one can check the services mountpoints.
E.g. with that we see the issue present in Artful before this upgrade.
$ findmnt -N $(systemctl status open-vm-tools | awk '/Main PID/ {print $3}')
After the upgrade we have
Artful: no private TMP (anymore, the version before this was affected)
Xenial: no private TMP (the upgrade did not add this issue since it is fixed before the backport)

Per above (and the extensive precheck on the content-equal ppa by VMWare) - setting to verified

tags: added: verification-done verification-done-artful verification-done-xenial
removed: verification-needed verification-needed-artful verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (10.0 KiB)

This bug was fixed in the package open-vm-tools - 2:10.2.0-3~ubuntu0.16.04.1

---------------
open-vm-tools (2:10.2.0-3~ubuntu0.16.04.1) xenial; urgency=medium

  * backport Bionic open-vm-tools to Xenial (LP: #1741390)
    - d/control: B-D for dh-autoreconf and dh-systemd
    - d/rules: re-add autoreconf and systemd
    - d/control: B-D to debhelper version of xenial supporting compat 10
    - d/compat: compat level 10 is latest xenial supported level
    - d/rules: go back from override_dh_installsystemd to
      override_dh_systemd_enable and override_dh_systemd_start as needed on
      xenials debhelper version.
    - d/rules: drop --no-restart-after-upgrade which only exists in debhelper
      10, the version 9 behavior defaults to what we need.
    - debian/rules: dh_systemd_start in xenial has issues with the escaping of
      the run-vmblock\\x2dfuse.mount job, so drop this call
    - debian/open-vm-tools-desktop.postinst: add a fixed version of what
      dh_systemd_start would have added
    - d/open-vm-tools-desktop.prerm, d/open-vm-tools-desktop.postrm: add what
      dh_systemd_start would have added (as-is since those sections worked)
    - d/control: update maintainers
  * d/open-vm-tools.service: Add After=local-fs.target dependency ensuring
    filesystems are ready to fix a race on startup (LP: #1750780)

open-vm-tools (2:10.2.0-3ubuntu3) bionic; urgency=medium

  * Disable PrivateTmp for the open-vm-tools.service as it triggers issues
    when triggering processes that need tmp through VMOMI API (LP: #1758428)

open-vm-tools (2:10.2.0-3ubuntu2) bionic; urgency=medium

  * Revert change in d/open-vm-tools.service that added After=local-fs.target.
    It turned out that the systemd in bionic already implicitly fixes
    this issue (the change is still needed for backports) (LP: 1750780)

open-vm-tools (2:10.2.0-3ubuntu1) bionic; urgency=medium

  * d/open-vm-tools.service: Add After=local-fs.target dependency ensuring
    filesystems are ready to fix a race on startup (LP: #1750780)

open-vm-tools (2:10.2.0-3) unstable; urgency=medium

  * [47e50a1] Fix debhelper dep for backports
  * [34538a5] Make tools.conf useful.
    Thanks to Dariusz Gadomski (Closes: #889884)

open-vm-tools (2:10.2.0-2) unstable; urgency=medium

  * [249d54c] Fix wayland segfault.
    Adding a patch from Fedora to fix a wayland/gnome related segfault.
    Thanks to Oliver Kurth (Closes: #887755)

open-vm-tools (2:10.2.0-1) unstable; urgency=medium

  * [f0bf956] Add .travis.yml which was removed by gbp.
  * [892e2f6] Build with gtk3.
  * [03a655b] Check if debhelper handles \ in systemd units.
    Thanks to Oliver Kurth (Closes: #886191)
  * [5bf9301] Drop -dkms package.
    Thanks to Christian Ehrhardt (Closes: #884656)
  * [236cdba] Update upstream source from tag 'upstream/10.2.0'
    Update to upstream version '10.2.0'
    with Debian dir d5190e486b6beb65ee7ed31c0c23a789b8f60cab
    (Closes: #884496)
  * [692beff] snapshot changelog.
  * [45aa743] Add .travis.yml which was removed by gbp.
  * [aabeded] Fix dpkg --compare-versions call
  * [0697425] Better debhelper version parsing.
  * [390ec09] fix even more makefile bugs.
  * [6e5fa38] Refreshing...

Changed in open-vm-tools (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for open-vm-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.7 KiB)

This bug was fixed in the package open-vm-tools - 2:10.2.0-3~ubuntu0.17.10.1

---------------
open-vm-tools (2:10.2.0-3~ubuntu0.17.10.1) artful; urgency=medium

  * backport Bionic open-vm-tools to Artful (LP: #1741390)
    - d/control: B-D for dh-autoreconf and dh-systemd
    - d/rules: re-add autoreconf and systemd
    - d/control: B-D to debhelper version of xenial supporting compat 10
    - d/compat: compat level 10 is latest xenial supported level
    - d/rules: go back from override_dh_installsystemd to
      override_dh_systemd_enable and override_dh_systemd_start as needed on
      xenials debhelper version.
    - d/rules: drop --no-restart-after-upgrade which only exists in debhelper
      10, the version 9 behavior defaults to what we need.
    - debian/rules: dh_systemd_start in xenial has issues with the escaping of
      the run-vmblock\\x2dfuse.mount job, so drop this call
    - debian/open-vm-tools-desktop.postinst: add a fixed version of what
      dh_systemd_start would have added
    - d/open-vm-tools-desktop.prerm, d/open-vm-tools-desktop.postrm: add what
      dh_systemd_start would have added (as-is since those sections worked)
    - d/control: update maintainers
  * d/open-vm-tools.service: Add After=local-fs.target dependency ensuring
    filesystems are ready to fix a race on startup (LP: #1750780)

open-vm-tools (2:10.2.0-3ubuntu3) bionic; urgency=medium

  * Disable PrivateTmp for the open-vm-tools.service as it triggers issues
    when triggering processes that need tmp through VMOMI API (LP: #1758428)

open-vm-tools (2:10.2.0-3ubuntu2) bionic; urgency=medium

  * Revert change in d/open-vm-tools.service that added After=local-fs.target.
    It turned out that the systemd in bionic already implicitly fixes
    this issue (the change is still needed for backports) (LP: 1750780)

open-vm-tools (2:10.2.0-3ubuntu1) bionic; urgency=medium

  * d/open-vm-tools.service: Add After=local-fs.target dependency ensuring
    filesystems are ready to fix a race on startup (LP: #1750780)

open-vm-tools (2:10.2.0-3) unstable; urgency=medium

  * [47e50a1] Fix debhelper dep for backports
  * [34538a5] Make tools.conf useful.
    Thanks to Dariusz Gadomski (Closes: #889884)

open-vm-tools (2:10.2.0-2) unstable; urgency=medium

  * [249d54c] Fix wayland segfault.
    Adding a patch from Fedora to fix a wayland/gnome related segfault.
    Thanks to Oliver Kurth (Closes: #887755)

open-vm-tools (2:10.2.0-1) unstable; urgency=medium

  * [f0bf956] Add .travis.yml which was removed by gbp.
  * [892e2f6] Build with gtk3.
  * [03a655b] Check if debhelper handles \ in systemd units.
    Thanks to Oliver Kurth (Closes: #886191)
  * [5bf9301] Drop -dkms package.
    Thanks to Christian Ehrhardt (Closes: #884656)
  * [236cdba] Update upstream source from tag 'upstream/10.2.0'
    Update to upstream version '10.2.0'
    with Debian dir d5190e486b6beb65ee7ed31c0c23a789b8f60cab
    (Closes: #884496)
  * [692beff] snapshot changelog.
  * [45aa743] Add .travis.yml which was removed by gbp.
  * [aabeded] Fix dpkg --compare-versions call
  * [0697425] Better debhelper version parsing.
  * [390ec09] fix even more makefile bugs.
  * [6e5fa38] Refreshing...

Read more...

Changed in open-vm-tools (Ubuntu Artful):
status: Fix Committed → Fix Released
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

After a serious dedup/crossreading session (so confusing :-/) I realized this needs to be reported to Debian, I did so and will link up the bug here ...

Changed in open-vm-tools (Debian):
status: Unknown → New
Changed in open-vm-tools (Debian):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.