[FFe] Add Modaliases to open-vm-tools-desktop to allow automatic installation by ubuntu-drivers

[Jean-Baptiste Lallement]

[Rationale]
On Ubuntu Desktop, we want to install automatically at installation time open-vm-tools-desktop and its dependencies to provide an improved user experience on first boot after installation of the guest. This method also provides an easy way to install the drivers after installation if they have not been installed at installation time.

It is achieved adding an XB-Modaliases field to the control file for the record 'open-vm-tools-desktop' so ubuntu-drivers can detect the packages to install when running in a VMware guest and adding open-vm-tools to the list of whitelisted packages in ubuntu-drivers

[Impact]
The change adds an additional field to the control file used by ubuntu-drivers to detect the drivers to install automatically. In the worst case scenario either the field is not added or the detection by ubuntu-drivers fails. In both cases the package o-v-t-desktop is not installed and can still be installed manually.

Note: open-vm-tools-desktop is in universe and for the final feature (>=Disco) it must be in main (already promoted). But for the SRU context (as this was accepted by Debian in 10.3.5-8) in our regular MRE we only cover the open-vm-tools change to "provide" the modalias. We do not change the ubuntu-drivers or ubiquity to depend on it - so no need to promote it on the SRU.

[Test Case]
With the patched versions of ubuntu-drivers and open-vm-tools
1. Boot a daily ISO of Ubuntu Desktop to the live session in a VMware Guest (with VMWare Workstation for example)
2. Open a terminal, make sure the package cache is up to date with the version of o-v-t containing the Modaliases field
3. Run the command:
       $ ubuntu-drivers list
   and verify that o-v-t-desktop is in the output
4. Run the command:
       $ ubuntu-drivers devices
   And verify that the output shows the details of the sys entry for this device
5. Run the command:
       $ sudo ubuntu-drivers install
   and verify that the package o-v-t-desktop is installed

[Regression Potential]
Very Low. Package may fail to build if for some reason dh-gencontrol fails. Otherwise, given that the package is not already installed automatically, if automated installation doesn't work then we're still in the same situation than today.
On ubuntu-drivers side, the change just adds o-v-t-desktop to the whitelist using an existing mechanism so other than a typo there is no risk of regression.

ProblemType: Bug
DistroRelease: Ubuntu 19.04
Package: open-vm-tools-desktop (not installed)
ProcVersionSignature: Ubuntu 4.19.0-13.14-generic 4.19.20
Uname: Linux 4.19.0-13-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.10-0ubuntu23
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Fri Mar 8 20:02:49 2019
InstallationDate: Installed on 2014-07-15 (1697 days ago)
InstallationMedia: Ubuntu 14.10 "Utopic Unicorn" - Alpha amd64 (20140520)
SourcePackage: open-vm-tools
UpgradeStatus: Upgraded to disco on 2018-03-24 (349 days ago)

[Jean-Baptiste Lallement (jibel)]

proposed change.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "open-vm-tools_10.3.5-7_10.3.5-7ubuntu1.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Jean-Baptiste Lallement (jibel)]

The updated package is available from my ppa: ppa:jibel/ppa

Here is the result with this ppa enabled:
root@ubuntu:~# apt policy open-vm-tools-desktop
open-vm-tools-desktop:
  Installed: (none)
  Candidate: 2:10.3.5-7ubuntu1
  Version table:
     2:10.3.5-7ubuntu1 500
        500 http://ppa.launchpad.net/jibel/ppa/ubuntu disco/main amd64 Packages

root@ubuntu:~# apt show open-vm-tools-desktop
Package: open-vm-tools-desktop
[...]
Modaliases: vmwgfx(pci:v000015ADd00000405sv*sd*bc*sc*i*)
[...]

root@ubuntu:~# ubuntu-drivers list
open-vm-tools-desktop

root@ubuntu:~# ubuntu-drivers devices
== /sys/devices/pci0000:00/0000:00:0f.0 ==
modalias : pci:v000015ADd00000405sv000015ADsd00000405bc03sc00i00
vendor : VMware
model : SVGA II Adapter
manual_install: True
driver : open-vm-tools-desktop - third-party free

[Christian Ehrhardt  (paelzer)]

Hi Jean-Baptiste,
I like the change and the automatism it will bring, but we have had some efforts to get Debian and us in Sync and this seems a non resolvable Delta for now.
The Debian Maintainer is smart and fast and therefore it was always great for us to have this package being a sync.

In this particular case dh_modalias [1] and co are a Ubuntu only thing for now - so we can't bring it there as-is. I wonder if you could also build a version using [2] for the d/rules changes.
Unfortunately I have no great suggestion for the d/control change as dh_moalias doesn#t exist in Debian at all. [3] as a similar discussion didn't lead to a "this is hwo you do it" and while [4] is working it confused enough people that I'd appreciate a smoother solution for anything implemented new (unless there is no other way).

I'd appreciate if you'd at least try to find a way to make this Debian compatible to drop the Delta in the long run (maybe you know there is a post-buster dh_modalias activity in Debian or anything like it - or you know of other such cases and how they were resolved?)

That said, I don't want to too be negative here. I reviewed the change/builds/ppa and I'm +1 on the suggested change for Ubuntu for the time being. You'll have to make this an FFE [5] bug to be allowed to upload it thou.
Will you do that Jean-Baptiste or do you need me to take over here?

[1]: http://manpages.ubuntu.com/manpages/bionic/man1/dh_modaliases.1.html
[2]: https://wiki.debian.org/DebVendorVariable
[3]: https://lists.debian.org/debian-mentors/2012/02/msg00710.html
[4]: https://salsa.debian.org/qemu-team/qemu/blob/debian-unstable/debian/rules#L290
[5]: https://wiki.ubuntu.com/FreezeExceptionProcess

[Jean-Baptiste Lallement (jibel)]

Patch to whitelist ubuntu-drivers-common

[Jean-Baptiste Lallement (jibel)]

PR for ubuntu-drivers-common

https://github.com/tseliot/ubuntu-drivers-common/pull/29/commits/a462c05e04001efca3d3fecf5c195308b250dfee

[Jean-Baptiste Lallement (jibel)]

debdiff for open-vm-tools against the package in Ubuntu. It overrides dh-gencontrol instead of using dh-modaliases which is Ubuntu specific so it can be integrated into upstream packaging.

The PR has been proposed upstream: https://github.com/bzed/pkg-open-vm-tools/pull/20

[Iain Lane (laney)]

OK for the FFe - please could you work with Christian to get this uploaded / upstream / etc in the best way?

Thanks :>

[Christian Ehrhardt  (paelzer)]

[08:14] <cpaelzer> jibel: we got the FFe on 1819207 and the change into Debian
[08:14] <cpaelzer> jibel: the only other change in Debian we want as well (an evolved version of a fix I already had tested and followed the upstream resolution)
[08:15] <cpaelzer> jibel: that means we now want to do a syncpackage - should I or do you want to?
[08:17] <jibel> hi cpaelzer , can you do it please?

I kicked the syncpackage and we can later on the day check if it all built and migrated as expected.

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-vm-tools - 2:10.3.5-8

---------------
open-vm-tools (2:10.3.5-8) unstable; urgency=medium

  [ Jean-Baptiste Lallement ]
  * [0f35aee] Add modaliases to open-vm-tools-desktop.
    Added Modaliases to open-vm-tools-desktop to auto-discover and
    auto-install the driver on Ubuntu via ubuntu-drivers. The driver is then
    installed at installation time and available on first boot for an
    improved user experience (LP: #1819207)

  [ Bernd Zeimetz ]
  * [dc4e1ce] Load vmwgfx module before vmtoolsd starts.
    As discussed on github in vmware/open-vm-tools#214
    we need to load the vmwgfx module before starting vmtoolsd
    for desktop users. Otherwise it is not able to retrieve the KMS
    resolutions and resizing the VM desktop fails.
    Thanks to @thomashvmw @rhertzog (Closes: #924518)

 -- Bernd Zeimetz <email address hidden> Wed, 13 Mar 2019 22:15:22 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-drivers-common - 1:0.6.3

---------------
ubuntu-drivers-common (1:0.6.3) disco; urgency=medium

  [ Jean-Baptiste Lallement ]
  * ubuntu-drivers,
    UbuntuDrivers/detect.py,
    tests/ubuntu_drivers.py:
    - Add a --free-only parameter, so as to allow installing only the
     available free drivers (LP: #1819207).
    - Add support for testing against a fake archive with free and
      non-free components.

 -- Alberto Milone <email address hidden> Mon, 18 Mar 2019 16:20:49 +0100

[Brian Murray (brian-murray)]

The ubiquity change was merged and uploaded.

[Christian Ehrhardt  (paelzer)]

The open-vm-tools portion of this overall change will be part of the regular backports for the package. But afaik, the other two will not be updated e.g. not in 18.04.x installer.
Marking tasks accordingly to reflect what the SRU team expects.

[Steve Langasek (vorlon)]

Hello Jean-Baptiste, or anyone else affected,

Accepted open-vm-tools into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-vm-tools/2:10.3.10-1~ubuntu0.18.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Jean-Baptiste, or anyone else affected,

Accepted open-vm-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-vm-tools/2:10.3.10-1~ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Christian Ehrhardt  (paelzer)]

While the main verification is on VMWare as usual this bug - since it is referenced in the changelog - is tracked as well.
And while right now there are no intentions to pick up the modalias with the other tools (e.g. installer) lets make sure this worked for the open-vm-tools portion on the backport.

I poked at ubunut-drivers a bit to check if things would work and got
open-vm-tools-desktop: installed: 2:10.3.10-1~ubuntu0.18.04.1 available: 2:10.3.10-1~ubuntu0.18.04.1 (auto-install) [distro] free
Not perfect, but ok - and given that we don't SRU the installer part that is ok.

For the context here the important part is if the modalias is there, and that is true for both releases.

$ apt show open-vm-tools-desktop 2>/dev/null | grep Modalias
Modaliases: vmwgfx(pci:v000015ADd00000405sv*sd*bc*sc*i*)

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for open-vm-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-vm-tools - 2:10.3.10-1~ubuntu0.18.04.1

---------------
open-vm-tools (2:10.3.10-1~ubuntu0.18.04.1) bionic; urgency=medium

  * Backport recent open-vm-tools (LP: #1822204)

open-vm-tools (2:10.3.10-1) unstable; urgency=high

  * [122e511] Update upstream source from tag 'upstream/10.3.10'
    Update to upstream version '10.3.10'
    with Debian dir fb12c7cfc99a9497795475c29306e78d08cc3712
    - Closes: #925940
    - Bugfix release for the 10.3 series.
      - Correct and/or improve handling of certain quiesced
        snapshot failures (shipped as patch in 2:10.3.5-6).
      - Fix some bad derefs in primary NIC gather code
      - Fix possible security issue with the permissions of the
        intermediate staging directory and path.
        Closes: #925959
      - CONSTANT_EXPRESSION_RESULT in TimeUtil_StringToDate()
        Found by coverity.
      - Deploypkg log files of linux should not be world readable.
        They might contain sensitive data.
      - General code clean-up:
        - Treat local variables "len" consistently as "size_t"
          type in Posix_Getmntent_r()
        - Improve readability of error handling logic in
          ShrinkDoWipeAndShrink() and remove another line of dead code.
        - Setting "errno" to ENOENT when there is no passwd entry
          for the user.
      - Fix NULL pointer dereference and remove three lines of dead code.
    - Other changes/fixes, not related to Debian:
      - Update copyright years
      - Fix CentOS 7.6 detection
      - Include vmware/tools/log.h to define g_info (fix for SLES)
      - Special-case profile loading for StartProgram
        (Win32 only)
      - Changes to common source files not applicable to
        open-vm-tools. (Code used by other vmware tools, unrelated
        to open-vm-tools).
      - Bump up the SYSIMAGE_VERSION for VMware tools 10.3.10

  * [18de70f] Removing backported patches, shipped in 10.3.10.

open-vm-tools (2:10.3.5-8) unstable; urgency=medium

  [ Jean-Baptiste Lallement ]
  * [0f35aee] Add modaliases to open-vm-tools-desktop.
    Added Modaliases to open-vm-tools-desktop to auto-discover and
    auto-install the driver on Ubuntu via ubuntu-drivers. The driver is then
    installed at installation time and available on first boot for an
    improved user experience (LP: #1819207)

  [ Bernd Zeimetz ]
  * [dc4e1ce] Load vmwgfx module before vmtoolsd starts.
    As discussed on github in vmware/open-vm-tools#214
    we need to load the vmwgfx module before starting vmtoolsd
    for desktop users. Otherwise it is not able to retrieve the KMS
    resolutions and resizing the VM desktop fails.
    Thanks to @thomashvmw @rhertzog (Closes: #924518)

 -- Christian Ehrhardt <email address hidden> Tue, 14 May 2019 09:07:32 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package open-vm-tools - 2:10.3.10-1~ubuntu0.18.10.1

---------------
open-vm-tools (2:10.3.10-1~ubuntu0.18.10.1) cosmic; urgency=medium

  * Backport recent open-vm-tools (LP: #1822204)

open-vm-tools (2:10.3.10-1) unstable; urgency=high

  * [122e511] Update upstream source from tag 'upstream/10.3.10'
    Update to upstream version '10.3.10'
    with Debian dir fb12c7cfc99a9497795475c29306e78d08cc3712
    - Closes: #925940
    - Bugfix release for the 10.3 series.
      - Correct and/or improve handling of certain quiesced
        snapshot failures (shipped as patch in 2:10.3.5-6).
      - Fix some bad derefs in primary NIC gather code
      - Fix possible security issue with the permissions of the
        intermediate staging directory and path.
        Closes: #925959
      - CONSTANT_EXPRESSION_RESULT in TimeUtil_StringToDate()
        Found by coverity.
      - Deploypkg log files of linux should not be world readable.
        They might contain sensitive data.
      - General code clean-up:
        - Treat local variables "len" consistently as "size_t"
          type in Posix_Getmntent_r()
        - Improve readability of error handling logic in
          ShrinkDoWipeAndShrink() and remove another line of dead code.
        - Setting "errno" to ENOENT when there is no passwd entry
          for the user.
      - Fix NULL pointer dereference and remove three lines of dead code.
    - Other changes/fixes, not related to Debian:
      - Update copyright years
      - Fix CentOS 7.6 detection
      - Include vmware/tools/log.h to define g_info (fix for SLES)
      - Special-case profile loading for StartProgram
        (Win32 only)
      - Changes to common source files not applicable to
        open-vm-tools. (Code used by other vmware tools, unrelated
        to open-vm-tools).
      - Bump up the SYSIMAGE_VERSION for VMware tools 10.3.10

  * [18de70f] Removing backported patches, shipped in 10.3.10.

open-vm-tools (2:10.3.5-8) unstable; urgency=medium

  [ Jean-Baptiste Lallement ]
  * [0f35aee] Add modaliases to open-vm-tools-desktop.
    Added Modaliases to open-vm-tools-desktop to auto-discover and
    auto-install the driver on Ubuntu via ubuntu-drivers. The driver is then
    installed at installation time and available on first boot for an
    improved user experience (LP: #1819207)

  [ Bernd Zeimetz ]
  * [dc4e1ce] Load vmwgfx module before vmtoolsd starts.
    As discussed on github in vmware/open-vm-tools#214
    we need to load the vmwgfx module before starting vmtoolsd
    for desktop users. Otherwise it is not able to retrieve the KMS
    resolutions and resizing the VM desktop fails.
    Thanks to @thomashvmw @rhertzog (Closes: #924518)

 -- Christian Ehrhardt <email address hidden> Tue, 14 May 2019 09:05:46 +0200

[Chris Halse Rogers (raof)]

An upload of ubuntu-drivers-common to bionic-proposed has been rejected from the upload queue for the following reason: "Includes a bunch of (hopefully!) non-functional whitespace changes. Changelog links to LP:# 1819207, marked as Won't Fix, and doesn't match the description in the changelog.".

[Christian Ehrhardt  (paelzer)]

Hmm, I know of nothing that was going on @jibel was that an upload by you?

[Alberto Milone (albertomilone)]

@Christian Ehrhardt: I was trying to backport the code for ubuntu-drivers-common, since it is tied to other changes that we need in bionic. The rest is not going to be backported.

[Steve Langasek (vorlon)]

Hello Jean-Baptiste, or anyone else affected,

Accepted ubuntu-drivers-common into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-drivers-common/1:0.5.2.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Alberto Milone (albertomilone)]

u-d-c in bionic-proposed works as expected. The --free-only feature doesn't show any candidates, also as expected, since only the u-d-c side was backported. The other features work as usual.

:~$ sudo ubuntu-drivers install --free-only
No drivers found for installation

:~$ apt-cache policy ubuntu-drivers-common
ubuntu-drivers-common:
  Installed: 1:0.5.2.4
  Candidate: 1:0.5.2.4
  Version table:
 *** 1:0.5.2.4 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1:0.5.2.3 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     1:0.5.2 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-drivers-common - 1:0.5.2.4

---------------
ubuntu-drivers-common (1:0.5.2.4) bionic; urgency=medium

  [ Alberto Milone ]
  * UbuntuDrivers/detect.py,
    UbuntuDrivers/kerneldetection.py,
    tests/ubuntu_drivers.py,
    ubuntu-drivers:
    - Provide DKMS fallback for the "install --gpgpu" case
      when no linux-modules-nvidia are available for the driver
      and for a specific kernel (LP: #1843796).
      We have new tests to cover this scenario now.
    - Show provider of kernel modules in list --gpgpu
    - Fix linux metapackage detection.
      This was meant to be driven by the latest kernel version
      But it was only partially implemented.
      Complete the implementation and update the relevant tests.
    - Make sure to install the linux-modules-nvidia package for the
      kernel flavour in use, when passing the --gpgpu argument, so
      that we can deliver a working NVIDIA installation, provided
      that the EULA was accepted.

  [ Jean-Baptiste Lallement ]
  * ubuntu-drivers,
    UbuntuDrivers/detect.py,
    tests/ubuntu_drivers.py:
    - Add a --free-only parameter, so as to allow installing only the
     available free drivers. This is a backport of only the
     ubuntu-drivers-common specific part of LP: #1819207, since
     it is tied to the rest of the changes to the test suite.
    - Add support for testing against a fake archive with free and
      non-free components.
    - Add static code checker.
  * debian/control:
    - Add pycodestyle and pyflake8 to build depends.

 -- Alberto Milone <email address hidden> Tue, 17 Sep 2019 16:34:56 +0200