Path of open-vm-tools libdeployPkgPlugin.so is now multi-arch compliant breaking cloud-init

[vmware-gos-Yuhua]

[Impact]

 * The package correctly fixed a non multiarch path, but we missed
   that some depending SW might have mad assumptions on the old paths.
   One such SW is cloud-init which in certain cases now fails to detect
   and configure vmware correctly.

 * In the long run (next Debian and 22.04) we will keep only the new
   paths. But for 21.10 time is too short and even more so for the SRUs
   that we regularly do back to at least the last LTS.
   There we want to mitigate the impact by adding a compat link on the
   old path.

[Test Plan]

 * We need to configure cloud-init to check for VMware IVMF data and we
   will see that without the fix it is failing to be detected.

 * Set up Ubuntu in VMWare if you do not ahve any ESXi then VMWare Workstation player 16 for Ubuntu as trial from https://www.vmware.com/products/workstation-player/workstation-player-evaluation.html is enough.

# Reduce the ouput a bit for readability and make it not skip vmware
$ echo "datasource_list: [NoCloud, OVF]" < sudo tee /etc/cloud/cloud.cfg.d/99_test.cfg
$ echo "disable_vmware_customization: false" | sudo tee -a /etc/cloud/cloud.cfg.d/99_test.cfg
# Run ds-dentify with debug on
$ sudo DEBUG_LEVEL=5 DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force
# Check the result
$ cat /run/cloud-init/cloud.cfg

Bad case example:
...
Checking for datasource 'OVF' via 'dscheck_OVF'
Running on vmware but rpctool query returned 1: No value found
check for 'OVF' returned not-found[1]
found= maybe=
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 3554.80s] returning 1

And the result is:
$ cat /run/cloud-init/cloud.cfg
di_report:
  datasource_list: [ ]
  # reporting not found result. notfound=disabled.

Good case example:
...
Checking for datasource 'OVF' via 'dscheck_OVF'
Running on vmware but rpctool query returned 1: No value found
/etc/cloud/cloud.cfg.d/99_test.cfg set disable_vmware_customization to false
check for 'OVF' returned found
found=OVF maybe=
Found single datasource: OVF
[up 3357.93s] returning 0

And the result is:
$ cat /run/cloud-init/cloud.cfg
datasource_list: [ OVF, None ]

Note: VMware who spotted this will do a verification as well on this case.

[Where problems could occur]

 * Since we do not remove, but add a link (that exactly matches the
   formerly used path) I'm not too concerned. The issue I can think of
   would be e.g. security policies that prevent .so files to load through
   symlinks or anything like that. But in that case still the upload
   would not further degrade things, it would just not fix it.
   Test wise this is all about guest customizations and VMware plugin and
   VMware as usual will do checks for that when this is in verification.

[Other Info]

 * Down the road we still want to drop that path, it is only added now to
   temporarily mitigate such issues. Therefore we do NOT want to have that
   in 22.04 for a long time, and will most likely drop it there soon to
   spot further issues with it.
 * For the same reason I'm also not uploading it to Debian via
   https://salsa.debian.org/vmware-packaging-team/pkg-open-vm-tools/-/merge_requests/11 there the active release isn't affected yet
   and the next one shall go without (like 22.04)
 * But for active Ubuntu release which got the backport I'd want to
   SRU fix it despite also having the cloud-init fix later on, since we do
   not know which other SW might rely on that path.

------

Problem:
the path of plugin libdeployPkgPlugin.so of open-vm-tools is changed and guest customization will fail for ubuntu 21.10 beta image

Description:
Ubuntu 21.10 have new open-vm-tools 11.3.0. with this new open-vm-tools, the plugin libdeployPkgPlugin.so is put to directory /usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/.

In previous open-vm-tools version (such as 11.2.5), the the plugin libdeployPkgPlugin.so is put to directory /usr/lib/open-vm-tools/plugins/vmsvc/

The path change of plugin libdeployPkgPlugin.so will cause the guest customization failure.

[vmware-gos-Yuhua (yhzou)]

cloud-init 21.3-1 will search plugin libdeployPkgPlugin.so in the four paths:
  search_paths = (
    "/usr/lib/vmware-tools", "/usr/lib64/vmware-tools",
    "/usr/lib/open-vm-tools", "/usr/lib64/open-vm-tools")

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[John Wolfe (johnwvmw)]

Was the move of the open-vm-tools plugins from /usr/lib/open-vm-tools to /usr/lib/x86_64-linux-gnu/open-vm-tools an intentional change in the 11.3.0 packaging?

Would it be feasible to repackage 11.3.0, adding a symbolic link from /usr/lib/open-vm-tools to x86_64-linux-gnu/open-vm-tools ?

[Christian Ehrhardt  (paelzer)]

/usr/lib/x86_64-linux-gnu/open-vm-tools is the correct path and due to the enablement or arm64 as requested we had to take care that it uses the correct path. There it will be /usr/lib/aarch64-linux-gnu/open-vm-tools or something like that.

See:
https://wiki.debian.org/Multiarch/Implementation
https://wiki.debian.org/Multiarch/Tuples

A compat link (as it is rather late) for one release and mid term fixing anything that depends on the old might be way forward.
But please note that you yourself verified already the backport thereof, so this is not only in impish but also in all of >=20.04.

I wonder if we better fix any consuming programs, is it just cloud-init or are there more we need to adapt=

[vmware-gos-Yuhua (yhzou)]

The plugin libdeployPkgPlugin.so is used for guest customization and consumed by cloud-init. I think it is just cloud-init we need to adapt.

(Note: this changed path don't affect the guest customization with perl engine in vsphere)

[vmware-gos-Yuhua (yhzou)]

> John Wolfe (johnwvmw) wrote 10 hours ago:#3
> Was the move of the open-vm-tools plugins from /usr/lib/open-vm-tools to /usr/lib/x86_64-linux->. gnu/open-vm-tools an intentional change in the 11.3.0 packaging?

> Would it be feasible to repackage 11.3.0, adding a symbolic link from /usr/lib/open-vm-tools to >. x86_64-linux-gnu/open-vm-tools ?

test it and it works well when add a symbolic link from /usr/lib/open-vm-tools to /usr/lib/x86_64-linux-gnu/open-vm-tools

Thanks
Yuhua Zou

[Christian Ehrhardt  (paelzer)]

Ok, let us retain the old path with a compat link "for now" meaning 21.10 impish and then related backports. I'm also interested what Bzed thinks about it and will throw him a PR to comment on once I have implemented this.

But at the same time we need to spawn a cloud-init bug tasks on this to be able to work with the correct future paths for 22.04 at least.

[Christian Ehrhardt  (paelzer)]

For clarification @John.
It isn't only .so files and not only one.
Of the former non-arch .so files we now have the following in proper directories:

open-vm-tools-desktop
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmusr/libdesktopEvents.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmusr/libdndcp.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmusr/libresolutionSet.so

open-vm-tools-dev:
./usr/lib/x86_64-linux-gnu/libDeployPkg.a
./usr/lib/x86_64-linux-gnu/libDeployPkg.so -> libDeployPkg.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestStoreClient.a
./usr/lib/x86_64-linux-gnu/libguestStoreClient.so -> libguestStoreClient.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestlib.a
./usr/lib/x86_64-linux-gnu/libguestlib.so -> libguestlib.so.0.0.0
./usr/lib/x86_64-linux-gnu/libhgfs.a
./usr/lib/x86_64-linux-gnu/libhgfs.so -> libhgfs.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvgauth.a
./usr/lib/x86_64-linux-gnu/libvgauth.so -> libvgauth.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvmtools.a
./usr/lib/x86_64-linux-gnu/libvmtools.so -> libvmtools.so.0.0.0
./usr/lib/x86_64-linux-gnu/pkgconfig/
./usr/lib/x86_64-linux-gnu/pkgconfig/libDeployPkg.pc
./usr/lib/x86_64-linux-gnu/pkgconfig/vmguestlib.pc

open-vm-tools-sdmp:
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libgdp.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libguestStore.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libserviceDiscovery.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/get-connection-info.sh
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/get-listening-process-info.sh
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/get-listening-process-perf-metrics.sh
./usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/get-versions.sh

open-vm-tools:
./usr/lib/x86_64-linux-gnu/libDeployPkg.so.0 -> libDeployPkg.so.0.0.0
./usr/lib/x86_64-linux-gnu/libDeployPkg.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestStoreClient.so.0 -> libguestStoreClient.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestStoreClient.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestlib.so.0 -> libguestlib.so.0.0.0
./usr/lib/x86_64-linux-gnu/libguestlib.so.0.0.0
./usr/lib/x86_64-linux-gnu/libhgfs.so.0 -> libhgfs.so.0.0.0
./usr/lib/x86_64-linux-gnu/libhgfs.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvgauth.so.0 -> libvgauth.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvgauth.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvmtools.so.0 -> libvmtools.so.0.0.0
./usr/lib/x86_64-linux-gnu/libvmtools.so.0.0.0
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/common/libhgfsServer.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/common/libvix.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libappInfo.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libdeployPkgPlugin.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libguestInfo.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libpowerOps.so
./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libresolution...

[vmware-gos-Yuhua (yhzou)]

Hi Christian Ehrhardt
> let us retain the old path with a compat link "for now" meaning 21.10 impish and then related >backports.

Thanks very much.

> But at the same time we need to spawn a cloud-init bug tasks on this to be able to work with the > correct future paths for 22.04 at least.
I will give feedback to vmware and ask to spawn a cloud-init bug tasks.

Best regards
Yuhua Zou

[Christian Ehrhardt  (paelzer)]

As suggested #1 isn't easily doable as long as #2 is meant to be present (cant't be a symlink and a directory at the same time). And we can't be sure (unless confirmed) that
/usr/lib/$(DEB_HOST_MULTIARCH)/open-vm-tools/serviceDiscovery/scripts/ will never have compiled elements. So for now I'll hold #1 back to not make things even worse.

For #2 being the compat link that should (tm) be trivial I hope.

PR for open-vm-tools at
https://salsa.debian.org/vmware-packaging-team/pkg-open-vm-tools/-/merge_requests/11

[Christian Ehrhardt  (paelzer)]

Hi John and Yuhua,
I have prepared a test build based on the change that I proved for Bernd in Salsa.
Could you have a test if this resolves (for now) the issues you have found?

=> https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4673

From the test build log:
lrwxrwxrwx root/root 0 2021-09-27 07:37 ./usr/lib/open-vm-tools -> x86_64-linux-gnu/open-vm-tools

Upgrade-installs fine over the former version.

Installed it looks like:
root@i:~# ll /usr/lib/open-vm-tools
lrwxrwxrwx 1 root root 30 Sep 27 07:37 /usr/lib/open-vm-tools -> x86_64-linux-gnu/open-vm-tools/
root@i:~# dpkg -S /usr/lib/open-vm-tools
open-vm-tools: /usr/lib/open-vm-tools

[vmware-gos-Yuhua (yhzou)]

Hi Christian Ehrhardt,
Sure. Thanks very much.

[vmware-gos-Yuhua (yhzou)]

Hi Christian Ehrhardt,

> https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4673
With this ppa it fix the issue "The path change of plugin libdeployPkgPlugin.so will cause the guest customization failure".

test with images:
1) ubuntu 21.10 desktop beta image
2) ubuntu 21.10 live server beta image

Check items:
   1) install open-vm-tools 11.3.0 from ppa
   2) upgrade to open-vm-tools 11.3.0 from ppa
   3) install open-vm-tools-desktop 11.3.0 from ppa for ubuntu xx.xx desktop image
   4) tools service and VGAuth service are running when install / upgrade open-vm-tools
   5) tools service and VGAuth service are running after install / upgrade and reboot guestOS
      uninstall open-vm-tools
   6) check guestInfo when install/uninstall open-vm-tools
   7) check guest customization with cloud-init engine
   8) check guest customization with perl engine

Thanks
Yuhua Zou

[vmware-gos-Yuhua (yhzou)]

Hi John Wolfe,
in #13, Any more testcases are needed due to #8 ? Thanks

[John Wolfe (johnwvmw)]

Hi Yuhua,

Please confirm that guest customization using cloud-init does, in fact, work with with this ppa.

Also, while the SDMP plugin is locating the scripts based on the installation directory, it would not hurt to run any basic SDMP test(s) available to confirm that there are no issues associated with the Muliarch packaging changes.

I am checking with the SDMP team that the script directory path has not been used as an absolute path in other applications.

Thanks Yuhua,
John

[vmware-gos-Yuhua (yhzou)]

Hi John Wolfe
 > Please confirm that guest customization using cloud-init does, in fact, work with with this ppa.

Yes. guest customization using cloud-init works well with ppa https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4673

 > I am checking with the SDMP team that the script directory path has not been used as an absolute path in other applications.

OK. Thanks.

[Christian Ehrhardt  (paelzer)]

Thanks everyone, from the test build and PR review POV all is fine.

I'm waiting for John to confirm that SDMP is really happy, to avoid fixing this overall issue twice.

To be clear @John - I'm ok to move the SDMP scripts subdir to a non multiarch directory if you could confirm that this will never contain architecture specific files (built code = bad, arch neutral scripts are fine). So in your talks with the SDMP team you might clarify that and depending on the outcome we might then move:
/usr/lib/x86_64-linux-gnu/open-vm-tools/serviceDiscovery/scripts/
back to
/usr/lib/open-vm-tools/serviceDiscovery/scripts/

[John Wolfe (johnwvmw)]

Hi Christian,

We are certain that the SDMP plugin ./usr/lib/x86_64-linux-gnu/open-vm-tools/plugins/vmsvc/libserviceDiscovery.so can be found and loaded since all the other vmsvc and vmusr plugins are successfully handled.

We are waiting on the vROps team
  - to run some smoke test(s) that use the libserviceDiscovery.so
  - to confirm that serviceDiscovery/scripts/* absolute paths have not escape into other applications as happened in the cloud-init case.

If there is an escape, the patch for the symbolic link

     /usr/lib/open-vm-tools ==> /usr/lib/<arch-rt>/open-vm-tools

would compensate for any path escape while that escape is fixed down stream.

[Xiaofeng Wang (xiaofengw)]

Hi Christian,
I will change cloud-init codes to search libdeployPkgPlugin.so in /usr/lib/<multiarch name>/open-vm-tools directory and fix this issue. Do you know how to get the multiarch name (normalized GNU triplets) on Ubuntu and Debian? Is there a easy way to achieve this? Thanks.

[Christian Ehrhardt  (paelzer)]

@Xiaofeng the following command will report the current one matching your machine.
  $ dpkg-architecture --query DEB_HOST_MULTIARCH
If you run it without arguments you will see that there are more/other variables in case you need them.

[Christian Ehrhardt  (paelzer)]

@John
Thanks for the further checks, once you and the vROps team are done let me know If I shall continue as proposed in the PPA (just the compat link) or if we should generally also move serviceDiscovery/scripts/* out of the multiarch dir.

[John Wolfe (johnwvmw)]

@Christian

The SDMP team has confirmed that the only use of the serviceDiscovery/scripts path and contents is by the service discovery plugin. That base of that path used by the SDMP compilation and make install of the SDMP scripts is derived from $(libdir) which in turn is derived from the configure script results. There is no problem locating the scripts in the existing multi-arch library location.

They have also confirmed that the scripts path has not "escaped" outside of the open-vm-tools plugin. There will be no problem as was seen with cloud-init.

Still waiting for the vROps team smoke test results, but given the above I do not think that there will be any problems. I will let you know as soon as I hear anything.

[Christian Ehrhardt  (paelzer)]

Thank you - waiting for the vROps team reply then.
But given the feedback so far I think we are looking forward to upload the fix as proposed and in the PPA to Debian and Ubuntu.

[Christian Ehrhardt  (paelzer)]

Still waiting for a reply, considering how important this might be for some users I wanted to ping again for a response to be able to finalize this as proposed or to adapt based on the feedback.
@John - do you have any updates?

[John Wolfe (johnwvmw)]

I share your impatience. I have pinged the vROps team that is running the SDMP smoke test.

John Savanyo has been cc'ed on that e-mail.

I agree with your assessment that the fix as proposed will work. It would be nice for the confirmation.

[Christian Ehrhardt  (paelzer)]

Since we will slip final freeze of 21.10 not yet having that confirmation we should start preparing the SRU template which we will need later on anyway. Just to pre-eliminate any later stalls.

For that the biggest missing piece for me is the "how to reproduce" you said it affects customization via cloud-init on vmware guests.
It wasn't stated how a setup for repro for would look like yet.

I currently lack a system to test this, so I've drafted steps based on the code but would like to ask you @John to revise them. Mission - as always - is to keep them as simple as possible which e.g. implies to - if possible - not use advanced VMWare host configurations.

---

But from studying the code I see a few things that make me wonder which exact config is needed to reach the code that is affected by this. Here from ds-identify code.

 991 dscheck_OVF() {
 992 check_seed_dir ovf ovf-env.xml && return "${DS_FOUND}"

If an image was built placing data in /var/lib/cloud/seed/ovf/...
That should not happen in a fresh clear environment and will likely be stepped over.

 994 [ "${DI_VIRT}" = "none" ] && return ${DS_NOT_FOUND}

This is only an early exit, if it is vmware it continues to check the rest

 996 # Azure provides ovf. Skip false positive by dis-allowing.
 997 is_azure_chassis && return $DS_NOT_FOUND

Not important in our cases as we are not talking about Azure, will be stepped over

 999 ovf_vmware_transport_guestinfo && return "${DS_FOUND}"

If here vmware-rpctool "info-get guestinfo.ovfEnv" gets content it returns with DS_FOUND

1001 has_ovf_cdrom && return "${DS_FOUND}"

If cloud-init config data is presented via cdrom it returns here.

1003 ovf_vmware_guest_customization && return "${DS_FOUND}"

Only if it stepped over all of the above, no config CDrom, no data via vmware-rpctool then the code of "ovf_vmware_guest_customization" which has the issue with the new path will be ran.

1005 return ${DS_NOT_FOUND}

This is the final, nothing found return

---

WIP Steps to reproduce to improve on
1. set up a VMWare based Ubuntu guest
 1.1 does the user or the tooling have to set disable_vmware_customization ?
     something like
     echo "search,found=all,maybe=all,notfound=disabled,disable_vmware_customization=false" | sudo tee /etc/cloud/ds-identify.cfg

 1.2 do any other options/customizations matter?

2. in the Ubuntu guest run ds-identify and check if the output detected vmware correctly

 2.1 $ sudo /usr/lib/cloud-init/ds-identify

 2.2 $ cat /run/cloud-init/cloud.cfg

In the good case this looks like ???
In the bad case it fails to detect VMware and looks like ???

[Christian Ehrhardt  (paelzer)]

Furthermore I tried to check this in VMWare Workstation player 16

First of all as I'd expect systemd-detect-virt identified vmware already.
$ systemd-detect-virt
vmware

The code affected by this issue here is in "ovf_vmware_guest_customization".
That is only used in dscheck_OVF.
Therefore to streamline the run reduce config to just detect OVF in ds-identify.

$ echo "datasource_list: [NoCloud, OVF]" < sudo tee /etc/cloud/cloud.cfg.d/99_test.cfg

Running ds-identify in verbose more to check behavior.

As expected I see it pass the steps before and reach ovf_vmware_guest_customization.
And indeed it fails to detect the pluging and does an early exit.

Here with a bit of additional debug inside of ovf_vmware_guest_customization:
$ sudo DEBUG_LEVEL=5 DI_LOG=stderr /usr/lib/cloud-init/ds-identify --force
[up 2345.15s] ds-identify --force
policy loaded: mode=search report=false found=all maybe=all notfound=disabled
/etc/cloud/cloud.cfg.d/90_dpkg.cfg set datasource_list: [NoCloud, OVF]
DMI_PRODUCT_NAME=VMware Virtual Platform
DMI_SYS_VENDOR=VMware, Inc.
DMI_PRODUCT_SERIAL=VMware-56 4d 5c 48 27 3b 66 c7-18 31 7d 99 e7 30 40 7e
DMI_PRODUCT_UUID=485c4d56-3b27-c766-1831-7d99e730407e
PID_1_PRODUCT_NAME=unavailable
DMI_CHASSIS_ASSET_TAG=No Asset Tag
FS_LABELS=
ISO9660_DEVS=
KERNEL_CMDLINE=BOOT_IMAGE=/boot/vmlinuz-5.11.0-20-generic root=UUID=e6b98230-678e-4963-88a1-673f9e943acd ro find_preseed=/preseed.cfg auto noprompt priority=critical locale=en_US quiet
VIRT=vmware
UNAME_KERNEL_NAME=Linux
UNAME_KERNEL_RELEASE=5.11.0-20-generic
UNAME_KERNEL_VERSION=#21+21.10.1-Ubuntu SMP Wed Jun 9 15:08:14 UTC 2021
UNAME_MACHINE=x86_64
UNAME_NODENAME=ubuntu
UNAME_OPERATING_SYSTEM=GNU/Linux
DSNAME=
DSLIST=NoCloud OVF
MODE=search
ON_FOUND=all
ON_MAYBE=all
ON_NOTFOUND=disabled
pid=54738 ppid=54737
is_container=false
Checking for datasource 'NoCloud' via 'dscheck_NoCloud'
check for 'NoCloud' returned not-found[1]
Checking for datasource 'OVF' via 'dscheck_OVF'
DEBUG: dscheck_OVF: Check seed
DEBUG: dscheck_OVF: DI_VIRT
DEBUG: dscheck_OVF: is_azure_chassis
DEBUG: dscheck_OVF: ovf_vmware_transport_guestinfo
Running on vmware but rpctool query returned 1: No value found
DEBUG: dscheck_OVF: has_ovf_cdrom
DEBUG: dscheck_OVF: ovf_vmware_guest_customization
DEBUG: ovf_vmware_guest_customization: start check
DEBUG: scan plugin
check for 'OVF' returned not-found[1]
found= maybe=
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 2345.23s] returning 1

With the fix (from PPA https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4673) in place it finds the plugin as expected.

We want to also force vmware customization itself for this test which isn't the default.

$ echo "disable_vmware_customization: false" | sudo tee -a /etc/cloud/cloud.cfg.d/99_test.cfg

And then the result is in /etc/cloud-init/cloud.cfg

We will see the check say:
...
Checking for datasource 'OVF' via 'dscheck_OVF'
Running on vmware but rpctool query returned 1: No value found
/etc/cloud/cloud.cfg.d/99_test.cfg set disable_vmware_customization to false
check for 'OVF' returned found
found=OVF maybe=
Found single datasource: OVF
[up 3357.93s] returning 0

And the result is:
$ cat /...

[Christian Ehrhardt  (paelzer)]

SRU Template ready now, juts waiting for that final confirmation from vROps team to start the upload to 21.10 and the SRUs from there.

[Christian Ehrhardt  (paelzer)]

TBH @John - I'm tempted to do this upload to Impish now for all the inbound deadlines.
Chances are quite high that this will be the right solution and today (Oct 7th) is the final freeze for 21.10 (https://discourse.ubuntu.com/t/impish-indri-release-schedule/18540).

Since this issue is severe for deployment and open-vm-tools it is a release critical bug that needs to go in before the release is ready.

But also it is seeded, so uploading this later will force us to re-build the RC ISOs/Images and to re-start testing.

I can either
a) wait and surely break & reset the 21.10 release once we have that confirmation
or
b) upload this quite likely fix now, and only if we spot issues re-upload a modification

Please continue to push on the vrops team, but clearly (b) should in any case be better for everyone, especially if it is confirmed to be the right fix.

Therefore I'll mark the bug accordingly and do the upload ...

[Christian Ehrhardt  (paelzer)]

Uploaded, accepted by the release team and built at https://launchpad.net/ubuntu/+source/open-vm-tools/2:11.3.0-2ubuntu1
Now enters automated testing and hopefully migrating before the first images/ISOs are built.

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-vm-tools - 2:11.3.0-2ubuntu1

---------------
open-vm-tools (2:11.3.0-2ubuntu1) impish; urgency=medium

  * d/rules: provide a compat link for the old open-vm-tools
    library/plugin paths (LP: #1944946)

 -- Christian Ehrhardt <email address hidden> Thu, 07 Oct 2021 09:31:05 +0200

[Christian Ehrhardt  (paelzer)]

@John - ok that worked well, let us hope no later fixups are needed.
Before I go for SRU of this to the other releases with 11.3 i'm waiting for this final feedback by you and the vRops team.

[John Wolfe (johnwvmw)]

@Christian - Just got the test results.

The Service Discovery smoke test(s) PASSED. The multi-arch layout does not impact the SDMP plugin.

The Guest Customization team will be resolving the libdeployPkgPlugin check in the cloud-init project.
  - quickly, by adding the multi-arch paths to the list of paths to be searched for the .so
  - revise the guest customization availability check to not depend on the list of fixed paths.
    (longer range fix)

In either case going forward, the /usr/lib/open-vm-tools symbolic link should only be needed in 11.3.0 and 11.3.5 open-vm-tools. It should not be needed by OVT 12.0.0 and Ubuntu 22.04.

[Christian Ehrhardt  (paelzer)]

Thanks for the confirmations

> In either case going forward, the /usr/lib/open-vm-tools symbolic link
> should only be needed in 11.3.0 and 11.3.5 open-vm-tools.
> It should not be needed by OVT 12.0.0 and Ubuntu 22.04.

My intention was to drop it slightly earlier.
I wanted to only have it in 11.3.0, but already drop it in 11.3.5 for
next Debian and 22.04.

[John Wolfe (johnwvmw)]

I was being conservative, not being aware of:
  - when the path fix will be made to cloud-init
  - when the next release of cloud-init (with the fix) will be available.

Open-vm-tools 11.3.5 on Ubuntu and Debian w/o the /usr/lib/open-vm-tools symbolic link will have a dependency on the updated next release of cloud-init.

I have raised the priority of the internal PR to guest customization has for the cloud-init fix.

Please keep and eye out for a pull request.

[James Falcon (falcojr)]

cloud-init fix has been committed upstream via https://github.com/canonical/cloud-init/pull/1061

[Xiaofeng Wang (xiaofengw)]

The cloud-init fix is delivered. So next Debian and Ububutu 22.04 will pick up this change, is it right? Is there any action I need to do for this issue? Thanks.

[Christian Ehrhardt  (paelzer)]

Hi Xiaofeng,
cloud-init releases regularly across Ubuntu - so even older releases will pick up the change with the next round of cloud-init updates. Unless the plan has changed that would be in a few weeks [1] plus some time to make it into SRUs from there.

But in addition I think we should also SRU the mitigation we've added to 21.10 open-vm-tools to ensure there is no other (than cloud-init) SW broken by this. I'll therefore prep these SRUs despite a fix in cloud-init coming as well.

[1]: https://discourse.ubuntu.com/t/cloud-init-2021-release-schedule/20897

[Christian Ehrhardt  (paelzer)]

FYI MPs and PPA uploads for Focal/Hirsute uploaded:
F: https://code.launchpad.net/~paelzer/ubuntu/+source/open-vm-tools/+git/open-vm-tools/+merge/410021
H: https://code.launchpad.net/~paelzer/ubuntu/+source/open-vm-tools/+git/open-vm-tools/+merge/410022
PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4673

[Brian Murray (brian-murray)]

Hello vmware-gos-Yuhua, or anyone else affected,

Accepted open-vm-tools into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-vm-tools/2:11.3.0-2ubuntu0~ubuntu21.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello vmware-gos-Yuhua, or anyone else affected,

Accepted open-vm-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-vm-tools/2:11.3.0-2ubuntu0~ubuntu20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[vmware-gos-Yuhua (yhzou)]

The issue is fixed when check it in the following images.
 1) ubuntu 20.04.3 desktop GA (-proposed repo)
 2) ubuntu 21.04 desktop GA. (-proposed repo)
 3) ubuntu 21.10 desktop GA
 4) ubuntu 21.10 live server GA
 5) ubuntu 21.10 cloud image

[Łukasz Zemczak (sil2100)]

Okay, I'll release this conditionally as I see all the respective series have been probably tested, but please be sure to include more information regarding which testing has been performed and on which package versions exactly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-vm-tools - 2:11.3.0-2ubuntu0~ubuntu21.04.2

---------------
open-vm-tools (2:11.3.0-2ubuntu0~ubuntu21.04.2) hirsute; urgency=medium

  * d/rules: provide a compat link for the old open-vm-tools
    library/plugin paths (LP: #1944946)
    - d/open-vm-tools.postinst: handle upgrades from <11.3.0-2 in regard
      to the symlink

 -- Christian Ehrhardt <email address hidden> Tue, 12 Oct 2021 07:50:08 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for open-vm-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-vm-tools - 2:11.3.0-2ubuntu0~ubuntu20.04.2

---------------
open-vm-tools (2:11.3.0-2ubuntu0~ubuntu20.04.2) focal; urgency=medium

  * d/rules: provide a compat link for the old open-vm-tools
    library/plugin paths (LP: #1944946)
    - d/open-vm-tools.postinst: handle upgrades from <11.3.0-2 in regard
      to the symlink

 -- Christian Ehrhardt <email address hidden> Tue, 12 Oct 2021 07:53:54 +0200

[James Falcon (falcojr)]

This bug is believed to be fixed in cloud-init in version 21.4. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 21.4-0ubuntu1~22.04.1

---------------
cloud-init (21.4-0ubuntu1~22.04.1) jammy; urgency=medium

  * d/upstream/metadata: Change contact to James Falcon
  * d/cloud-init.templates: Add LXD to default datasource_list with
    translations
  * drop the following cherry-picks now included:
    + cpick-28e56d99-Azure-Retry-dhcp-on-timeouts-when-polling
    + cpick-e69a8874-Set-Azure-to-only-update-metadata-on-BOOT_NEW_INSTANCE
    + cpick-612e3908-Add-connectivity_url-to-Oracle-s-EphemeralDHCPv4-988
    + cpick-dc227869-Set-Azure-to-apply-networking-config-every-BOOT-1023
    + cpick-9c147e83-Allow-disabling-of-network-activation-SC-307-1048
  * New upstream release.
    - Release 21.4 (#1091) (LP: #1949405)
    - Azure: fallback nic needs to be reevaluated during reprovisioning
      (#1094) [Anh Vo]
    - azure: pps imds (#1093) [Anh Vo]
    - testing: Remove calls to 'install_new_cloud_init' (#1092)
    - Add LXD datasource (#1040)
    - Fix unhandled apt_configure case. (#1065) [Brett Holman]
    - Allow libexec for hotplug (#1088)
    - Add necessary mocks to test_ovf unit tests (#1087)
    - Remove (deprecated) apt-key (#1068) [Brett Holman] (LP: #1836336)
    - distros: Remove a completed "TODO" comment (#1086)
    - cc_ssh.py: Add configuration for controlling ssh-keygen output (#1083)
      [dermotbradley]
    - Add "install hotplug" module (SC-476) (#1069) (LP: #1946003)
    - hosts.alpine.tmpl: rearrange the order of short and long hostnames
      (#1084) [dermotbradley]
    - Add max version to docutils
    - cloudinit/dmi.py: Change warning to debug to prevent console display
      (#1082) [dermotbradley]
    - remove unnecessary EOF string in
      disable-sshd-keygen-if-cloud-init-active.conf (#1075) [Emanuele
      Giuseppe Esposito]
    - Add module 'write-files-deferred' executed in stage 'final' (#916)
      [Lucendio]
    - Bump pycloudlib to fix CI (#1080)
    - Remove pin in dependencies for jsonschema (#1078)
    - Add "Google" as possible system-product-name (#1077) [vteratipally]
    - Update Debian security suite for bullseye (#1076) [Johann Queuniet]
    - Leave the details of service management to the distro (#1074)
      [Andy Fiddaman]
    - Fix typos in setup.py (#1059) [Christian Clauss]
    - Update Azure _unpickle (SC-500) (#1067) (LP: #1946644)
    - cc_ssh.py: fix private key group owner and permissions (#1070)
      [Emanuele Giuseppe Esposito]
    - VMware: read network-config from ISO (#1066) [Thomas Weißschuh]
    - testing: mock sleep in gce unit tests (#1072)
    - CloudStack: fix data-server DNS resolution (#1004)
      [Olivier Lemasle] (LP: #1942232)
    - Fix unit test broken by pyyaml upgrade (#1071)
    - testing: add get_cloud function (SC-461) (#1038)
    - Inhibit sshd-keygen@.service if cloud-init is active (#1028)
      [Ryan Harper]
    - VMWARE: search the deployPkg plugin in multiarch dir (#1061)
      [xiaofengw-vmware] (LP: #1944946)
    - Fix set-name/interface DNS bug (#1058) [Andrew Kutz] (LP: #1946493)
    - Use specified tmp location for growpart (#1046) [jshen28]
    - .gitignore: ignore tags file for ctags users (#1057) [Brett Holman]
 ...

[Christian Ehrhardt  (paelzer)]

FYI - as planned (and now that cloud init is fixed) I'm planning to upload this to Jammy without the path fix. There should be no issue with that.

But planning ahead, there will later on be a backport of open-vm-tools 11.3.5 with that to >=Focal.
I checked
21.4-0ubuntu1~20.04.1
21.4-0ubuntu1~21.04.1
Both have the fix and therefor do not block the backport.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3909