plainbox-provider-certification-client wants to remove ubuntu-desktop and xorg from Precise 12.04.4

[Daniel Manrique]

I noticed after installing our tools, precise installations became borked and wouldn't boot to the graphical desktop. I ended up tracing this to the fact that a huge set of X-related packages, including xorg and ubuntu-desktop, are removed when installing plainbox-provider-certification-client.

I need to debug this further, just wanted to leave the report here so I don't forget the details.

To reproduce:

- Boot a 12.04.4 desktop live CD (I tried amd64), in "try ubuntu before installing" mode.
sudo apt-add-repository ppa:ubuntu-sdk-team/ppa
sudo apt-add-repository ppa:checkbox-dev/ppa
- Enable universe and multiverse, on Precise I think it has to be done by editing /etc/apt/sources.list.
apt-get update
apt-get install plainbox-provider-certification-client

Expected:
- Installation of provider and all its dependencies, without borking the system.
Actual:
- Huge set of packages marked for removal:
The following packages will be REMOVED:
  libegl1-mesa-drivers-lts-saucy libegl1-mesa-lts-saucy libgbm1-lts-saucy
  libgl1-mesa-dri-lts-saucy libgl1-mesa-glx-lts-saucy libglamor-ltss0
  libglapi-mesa-lts-saucy libopenvg1-mesa-lts-saucy libxatracker1-lts-saucy
  ubuntu-desktop x11-xserver-utils-lts-saucy xorg xserver-common-lts-saucy
  xserver-xorg-core-lts-saucy xserver-xorg-glamoregl-lts-saucy
  xserver-xorg-input-all-lts-saucy xserver-xorg-input-evdev-lts-saucy
  xserver-xorg-input-mouse-lts-saucy xserver-xorg-input-synaptics-lts-saucy
  xserver-xorg-input-vmmouse-lts-saucy xserver-xorg-input-wacom-lts-saucy
  xserver-xorg-lts-saucy xserver-xorg-video-all-lts-saucy
  xserver-xorg-video-ati-lts-saucy xserver-xorg-video-cirrus-lts-saucy
  xserver-xorg-video-fbdev-lts-saucy xserver-xorg-video-intel-lts-saucy
  xserver-xorg-video-mach64-lts-saucy xserver-xorg-video-mga-lts-saucy
  xserver-xorg-video-modesetting-lts-saucy
  xserver-xorg-video-neomagic-lts-saucy xserver-xorg-video-nouveau-lts-saucy
  xserver-xorg-video-openchrome-lts-saucy xserver-xorg-video-r128-lts-saucy
  xserver-xorg-video-radeon-lts-saucy xserver-xorg-video-s3-lts-saucy
  xserver-xorg-video-savage-lts-saucy
  xserver-xorg-video-siliconmotion-lts-saucy xserver-xorg-video-sis-lts-saucy
  xserver-xorg-video-sisusb-lts-saucy xserver-xorg-video-tdfx-lts-saucy
  xserver-xorg-video-trident-lts-saucy xserver-xorg-video-vesa-lts-saucy
  xserver-xorg-video-vmware-lts-saucy

Note these are from the Saucy enablement stack, that's most likely related.

I'll try installing the provider's depends and recommends to see which one is causing this.

[Daniel Manrique (roadmr)]

Interesting! if I install all the dependencies manually:

apt-get install bootchart fswebcam fwts glmark2 glmark2-es gtkperf hdparm imagemagick mesa-utils obexftp render-bench shutter sox stress wmctrl checkbox-ng-service plainbox-provider-checkbox plainbox-provider-resource-generic python3-plainbox

then I install the provider:

apt-get install plainbox-provider-certification-client

then the desktop install doesn't go borked.

What's going on :)

[Zygmunt Krynicki (zyga)]

Oddly enough. I cannot reproduce that. I have tried exactly your steps plus a number of variations. I wonder if that could have been a one-off fluke

[Zygmunt Krynicki (zyga)]

Marking as incomplete. Cannot reproduce in any way.

[Daniel Manrique (roadmr)]

Whaat, I just reproduced it very easily. Are you using a fully-installed 12.04.4 system? that may be a difference, I've seen this on systems that are in the process of being installed (e.g. a live CD environment), though I think there should be no difference. I'll try installing first and then trying to replicate this.

Anyway, I'm attaching a diagnostic log produced by apt-get --dry-run -o Debug::pkgProblemResolver=1 install plainbox-provider-certification-client.

[Daniel Manrique (roadmr)]

Marking as confirmed again as we have a dupe; also triaging as high because it renders checkbox-satellite unusable for 12.04.4 installations.

[Daniel Manrique (roadmr)]

OK, I think I found it:

  Installing glmark2-es2 as Recommends of plainbox-provider-certification-client
    Installing libgles2-mesa as Depends of glmark2-es2
      Installing libglapi-mesa as Depends of libgles2-mesa

this libglapi-mesa thingy is provided by libglapi-mesa-lts-saucy, which is part of the enablement stack. Relevant data from that package:

Package: libglapi-mesa-lts-saucy
Version: 9.2.1-1ubuntu3~precise1
Replaces: libglapi-mesa
Provides: libglapi-mesa, xorg-renamed-package, xorg-renamed-package-lts-saucy
Conflicts: libglapi-mesa, xorg-renamed-package-lts-quantal, xorg-renamed-package-lts-raring

As seen, it correctly replaces/conflicts the vanilla libglapi-mesa package, and provides libglapi-mesa, which should keep dependents happy.

However, our direct dependency (libgles2-mesa) is NOT happy with the 9.2.1 version of libglpi-mesa as provided by the above package. It has this:

Package: libgles2-mesa
Version: 8.0.2-0ubuntu3
Replaces: libgles2
Provides: libgles2
Depends: libglapi-mesa (= 8.0.2-0ubuntu3), libc6 (>= 2.2.5)
Conflicts: libgles2

So it wants a specific version of libglapi-mesa, thus causing:

- Replacement of libglapi-mesa-lts-saucy with libglapi-mesa (as seen in the attached problemresolver output)
- Installing libglapi-mesa on its own results on removal of the same packages shown in the original report, i.e. destroying the entire saucy enablement stack and ubuntu-desktop which depends on it.

When installing via plainbox-provider-certification-client, that's what happens. Next I'll look at why installing the dependencies manually doesn't cause things to break.

[Daniel Manrique (roadmr)]

OK, a suitable workaround is to install glmark2-es2 prior to installing plainbox-provider-certification-client. For some reason still to be determined, installing that first correctly finds the saucy-stack dependency:

$ sudo apt-get install glmark2-es2
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
  glmark2-data libgles2-mesa-lts-saucy
The following NEW packages will be installed:
  glmark2-data glmark2-es2 libgles2-mesa-lts-saucy
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.

then the install of the provider doesn't remove any packages:

$ sudo apt-get install plainbox-provider-certification-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  python-dateutil
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  bonnie++ bootchart checkbox-ng checkbox-ng-service curl dkms ethtool fakeroot fswebcam fwts fwts-efi-runtime-dkms glmark2 gtkperf heirloom-mailx imagemagick imagemagick-common libbfb0
  libbonobo2-0 libbonobo2-common libbonoboui2-0 libbonoboui2-common libcdt4 libcommon-sense-perl libencode-locale-perl libextutils-depends-perl libextutils-pkgconfig-perl
  libfile-listing-perl libfile-which-perl libfont-afm-perl libfwts1 libfwtsacpica1 libfwtsiasl1 libgif4 libglade2-0 libgnome2-0 libgnome2-bin libgnome2-canvas-perl libgnome2-gconf-perl
  libgnome2-perl libgnome2-vfs-perl libgnome2-wnck-perl libgnomecanvas2-0 libgnomecanvas2-common libgnomeui-0 libgnomeui-common libgnomevfs2-0 libgnomevfs2-common libgnomevfs2-extra
  libgoo-canvas-perl libgoocanvas-common libgoocanvas3 libgraph4 libgsm1 libgtk2-imageview-perl libgtk2-unique-perl libgtkimageview0 libgvc5 libhtml-form-perl libhtml-format-perl
  libhtml-parser-perl libhtml-tagset-perl libhtml-tree-perl libhttp-cookies-perl libhttp-daemon-perl libhttp-date-perl libhttp-message-perl libhttp-negotiate-perl libhttp-server-simple-perl
  libid3tag0 libidl-common libidl0 libilmbase6 libimlib2 libio-socket-inet6-perl libio-socket-ssl-perl libjson-perl libjson-xs-perl liblqr-1-0 liblwp-mediatypes-perl
  liblwp-protocol-https-perl libmagickcore4 libmagickcore4-extra libmagickwand4 libmailtools-perl libmulticobex1 libnet-dbus-perl libnet-http-perl libnet-ssleay-perl libnetpbm10 libobexftp0
  libopencore-amrnb0 libopencore-amrwb0 libopenexr6 liborbit2 libpath-class-perl libpathplan4 libproc-processtable-perl libproc-simple-perl libsocket6-perl libsort-naturally-perl
  libsox-fmt-alsa libsox-fmt-base libsox1b libtie-ixhash-perl libtimedate-perl libunique-1.0-0 liburi-perl libwww-mechanize-perl libwww-perl libwww-robotrules-perl libx11-protocol-perl
  libxml-namespacesupport-perl libxml-parser-perl libxml-sax-base-perl libxml-sax-expat-perl libxml-sax-perl libxml-simple-perl libxml-twig-perl libxml-xpath-perl netpbm nmap obexftp
  perlmagick plainbox-provider-checkbox plainbox-provider-resource-generic plainbox-secure-policy pybootchartgui python-support python3-chardet python3-checkbox-ng python3-checkbox-support
  python3-pk...

[Taihsiang Ho (tai271828)]

hi Daniel,
your quick workaround to install glmark2-es2 prior to plainbox-provider-certification-client
works for precise (lts-saucy-backport)
but not for precise (lts-quantal-backport) and precise (lts-raring-backport)

[Daniel Manrique (roadmr)]

Hm, probably with the other enablement stacks there are some more versioning problems.

I'll download 12.04.3 (raring) and 12.04.2 (quantal) to test, although it will take a bit longer because I need to download the full ISO images and my home link is not so fast.

Interesting note: the only test requiring glmark2-es2 is this one:

plugin: shell
id: benchmarks/graphics/glmark2-es2
requires:
 package.name == 'glmark2-es2'
 'arm' in cpuinfo.type
command:
 glmark2-es2 2>&1 | sed -e :a -e '$!N;s/\n/ /;ta' | sed -E 's/.*(Score:\s+[0-9]+).*/\1/'
_description: Run GLmark2-ES2 benchmark

and it's not on any of our whitelists.

[Daniel Manrique (roadmr)]

OK, I confirmed a similar but not exactly identical problem with 12.04.2 (quantal enablement stack).

I looked at the dependencies more closely, starting from the top-level glmark-es2 package, on this 12.04.2 system:

Package: glmark2-es2
Depends: libc6 (>= 2.4), libegl1-mesa (>= 7.8.1) | libegl1-x11, libgcc1 (>= 1:4.1.1), libgles2-mesa (>= 7.8.1) | libgles2, libpng12-0 (>= 1.2.13-4), libstdc++6 (>= 4.6), libx11-6, glmark2-data (= 2011.09-0ubuntu1)

These are interesting:

libegl1-mesa (>= 7.8.1) | libegl1-x11
ibgles2-mesa (>= 7.8.1) | libgles2

As mentioned above, libgles2-mesa is the one that breaks the X stack, by bringing in a specific version of libglapi-mesa:

Package: libgles2-mesa
Replaces: libgles2
Provides: libgles2
Depends: libglapi-mesa (= 8.0.2-0ubuntu3), libc6 (>= 2.2.5)

libegl1-mesa does something similar altough its dependency chain is a bit more obscure.

Now, notice that both dependencies have an alternative (| libgles2 for instance). This means that if a package providing 'libgles2' were installed, then libgles2-mesa wouldn't be installed and wouldn't break things. Which packages provide libgles2? A lot, most of them are the per-release enablement stacks. For instance:

Package: libgles2-mesa-lts-quantal
Replaces: libgles2, libgles2-mesa
Provides: libgles2, libgles2-mesa, xorg-renamed-package, xorg-renamed-package-lts-quantal

A similar situation with libegl1-mesa, packages providing libegl1-x11 include:

Package: libegl1-mesa-lts-quantal
Replaces: libegl1-mesa, libegl1-x11
Provides: libegl1-mesa, libegl1-x11, xorg-renamed-package, xorg-renamed-package-lts-quantal

So the problem is:

- If none of those lts-quantal packages are installed, glmark2-es2 will try to install the first packages listed (the ones on the left side of the | ), based on how debian package dependencies are processed. This breaks the stack on a freshly-installed system.

- But if I install the stack-specific packages first (libegl1-mesa-lts-quantal, libgles2-mesa-lts-quantal), then their provides: satisfy the requirements for glmark2-es2, which then doesn't install the breaker packages and everything works fine.

[Daniel Manrique (roadmr)]

OK, I looked at 12.04.4 more closely, and it's a similar but not identical problem. The default saucy install includes libegl1-mesa-lts-saucy, but does NOT include libgles2-mesa-lts-saucy. Interestingly, trying to install glmark2-es2 by itself pulls the correct libgles2-mesa-lts-saucy:

$ sudo apt-get install glmark2-es2
The following extra packages will be installed:
  glmark2-data libgles2-mesa-lts-saucy

On the resolver I can see it doesn't want to install libgles:

Investigating (0) libgles2-mesa [ amd64 ] < none -> 8.0.4-0ubuntu0.7 > ( libs )
Broken libgles2-mesa:amd64 Depends on libglapi-mesa [ amd64 ] < none -> 8.0.4-0ubuntu0.7 > ( libs ) (= 8.0.4-0ubuntu0.7)
  Considering libglapi-mesa:amd64 0 as a solution to libgles2-mesa:amd64 1
  Holding Back libgles2-mesa:amd64 rather than change libglapi-mesa:amd64

eventually it decides to install the lts package:

Investigating (5) glmark2-es2 [ amd64 ] < none -> 2011.09-0ubuntu1 > ( universe/misc )
Broken glmark2-es2:amd64 Depends on libgles2-mesa [ amd64 ] < none -> 8.0.4-0ubuntu0.7 > ( libs ) (>= 7.8.1)
  Considering libgles2-mesa:amd64 1 as a solution to glmark2-es2:amd64 9999
  Considering libgles2-mesa:amd64 1 as a solution to glmark2-es2:amd64 9999
  Considering libgles2-mesa:amd64 1 as a solution to glmark2-es2:amd64 9999
Broken glmark2-es2:amd64 Depends on libgles2 [ amd64 ] < none > ( none )
  Considering libgles2-mesa-lts-raring:amd64 54 as a solution to glmark2-es2:amd64 9999
  Considering libgles2-mesa-lts-quantal:amd64 54 as a solution to glmark2-es2:amd64 9999
  Considering libgles2-mesa:amd64 1 as a solution to glmark2-es2:amd64 9999
  Considering libgles2-mesa-lts-trusty:amd64 0 as a solution to glmark2-es2:amd64 9999
  Considering libgles2-mesa-lts-saucy:amd64 0 as a solution to glmark2-es2:amd64 9999
  Re-Instated libgles2-mesa-lts-saucy:amd64

but when trying to install via plainbox-provider-certification-client, it wants to install libgles2-mesa. Apparently very early on in the problem solving process it decides it's OK to install libgles2-mesa at the expense of removing xorg:

Investigating (0) xserver-xorg-lts-saucy [ amd64 ] < 1:7.7+1ubuntu6~precise1 > ( x11 )
Broken xserver-xorg-lts-saucy:amd64 Conflicts on libgles2-mesa [ amd64 ] < none -> 8.0.4-0ubuntu0.7 > ( libs ) (>= 0~)
  Considering libgles2-mesa:amd64 0 as a solution to xserver-xorg-lts-saucy:amd64 0
  Removing xserver-xorg-lts-saucy:amd64 rather than change libgles2-mesa:amd64
Investigating (0) libgles2-mesa [ amd64 ] < none -> 8.0.4-0ubuntu0.7 > ( libs )
Broken libgles2-mesa:amd64 Depends on libglapi-mesa [ amd64 ] < none -> 8.0.4-0ubuntu0.7 > ( libs ) (= 8.0.4-0ubuntu0.7)
  Considering libglapi-mesa:amd64 0 as a solution to libgles2-mesa:amd64 0
  Holding Back libgles2-mesa:amd64 rather than change libglapi-mesa:amd64

This looks like a very obscure apt resolving bug :/ and the problem is, it only happens when glmark2-es2 is installed as a dependency, not when installed directly.

[Daniel Manrique (roadmr)]

Interesting:

- On 12.04.4 and 12.04.3, both libegl1 and libgles2 from backports (saucy) are needed for the provider to install cleanly.
- On 12.04,2, only libegl1 from backports (quantal) is needed for the provider to install cleanly.

This is definitely a very subtle problem with package priorities as decided by apt-get. I think this is by the very nature of the enablement stacks; the Ubuntu CDs come with a pre-installed set of packages from one of the stacks, but I didn't find an easy way to "depend on package X from the enablement stack I have installed". I even saw different behaviors depending on whether I recommend or depend on the required packages (recommends usually tries to remove the packages, while depends usually complains about missing dependencies).

I tried something like:

Depends: libgles2-mesa-lts-saucy | libgles2-mesa-lts-raring | libgles2-mesa-lts-quantal | libgles2-mesa (>= 7.8.1),
         libegl1-mesa-lts-saucy | libegl1-mesa-lts-raring | libegl1-mesa-lts-quantal | libgles2-mesa (>= 7.8.1)

But even then, it doesn't really choose the appropriate set of packages; this only worked correctly on trusty and 12.04.4 (saucy stack), but fails (wants to remove packages in 12.04.2-quantal, and simply fails due to unmet dependencies on 12.04.3-raring).

As it stands, the only way to solve this reliably is to preinstall both libgles2-mesa and libegl1-mesa from the appropriate enablement stack prior to (or along with, in the same apt-get command) installation of our provider. The problem is that determining the appropriate version of the packages is a bit tricky.

I'll ask on #ubuntu-devel on Monday to see if there's a better way to handle this, and then update accordingly.

If we find no other way, then we can make glmark2-es2 an arch-specific dependency per https://www.debian.org/doc/debian-policy/ch-relationships.html, since it's only relevant on arm (and I'm not sure we care about backported enablement stacks on arm):

Depends: blah, blah, glmark2-es2 [armhf arm64]

this will avoid installing the dependency (and thus sidestep the entire conflict) on non-arm architectures, meaning the test will not run by virtue of its job definition's package requirements.

[Daniel Manrique (roadmr)]

Filed "upstream" as https://bugs.launchpad.net/ubuntu/+source/xorg-lts-transitional/+bug/1346409.

[Daniel Manrique (roadmr)]

OK, so the upstream bug is about the apt-get strange behavior, to avoid burying this problem in our stack of bugs :)

On our side, things are as follows:

- We will not be testing the last set of LTS kernels for 12.04.* in the next couple of weeks;
- After that set of kernels (in about 2 weeks), the 12.04.{2,3,4} stacks will go EOL and the only supported stack will be the trusty one from 12.04.5.

So we don't need to find a solution that works for 12.04.{2,3,4}, we just need something that works with 12.04.5. In this case, we can simply modify our preseeds to preinstall the libegl1 and libgles2 packages from the trusty stack and that should solve the problem. Optionally, we can check if glmark2-es2 can be preinstalled as it was in 12.04.4, and just do that.

This can be implemented in checkbox-satellite, so I'll mark this bug as won't fix for the provider (since we will NOT be changing our packaging for this), and leave a triaged task for checkbox-satellite to implement those preseed changes.

[Po-Hsu Lin (cypressyew)]

I tried to add apt-get install with these two packages in the late_command file,

=== modified file 'preseeds/precise/late_command'
--- preseeds/precise/late_command 2014-05-07 04:15:23 +0000
+++ preseeds/precise/late_command 2014-08-01 11:43:16 +0000
@@ -24,6 +24,7 @@
   echo 'deb <% certification_repository %> <% series %> main' >> /target/etc/apt/sources.list; \
 % endif
   in-target apt-get update; \
+ in-target apt-get install -y --force-yes libegl1-mesa libgles2-mesa; \
 % if not plainbox
 % if http_proxy
   export http_proxy=<% http_proxy %>; \

But it seems that I didn't manage to make it work
ubuntu@201208-11587:~$ dpkg -l | grep libegl
rc libegl1-mesa-lts-trusty 10.1.3-0ubuntu0.1~precise1 free implementation of the EGL API -- runtime
ubuntu@201208-11587:~$ dpkg -l | grep libgles2
ubuntu@201208-11587:~$

[Daniel Manrique (roadmr)]

THanks! But that's the entire problem: instead of installing the actual packages, you need to install the enablement stack packages that "provide" those virtual packages. So for 12.04.5 you have to install libegl1-mesa-lts-trusty libgles2-mesa-lts-trusty.

The problem comes from the fact that a different set of packages is needed for each point release: for 12.04.4 you need saucy packages, for 12.04.3, raring packages, and for 12.04.2, quantal packages. This is what's proving a bit difficult to express in the preseed.

Ara said that we can potentially ignore all point releases except .5, but we also need to keep 12.04 (no point) installation working; the problem here is that if I force installation of libegl1-mesa-lts-trusty on 12.04.0, it may try to install the entire enablement stack and destroy the installation again.

I'm prototyping a solution which "detects" the point release and installs the appropriate enablement stack packages, I promise I'll have something ready for review today.

[Daniel Manrique (roadmr)]

As it turns out, code to select the appropriate enablement stack wasn't that difficult to write.

The proposed solution works with all point releases :)