[SRU] No graphics after update, caused by virtualbox

[Akos Nagy]

[Impact]
* ubuntu when used as a guest inside a Virtualbox VM machine

[Test Case]
* Install Ubuntu and mate or other DM, experience a black screen

* Install vbox on xenial with guest and host. Test if everything works and starts with 3d enabled and disabled, as well as different Desktop Managers.

[ autopkgtests ]
* this package has a really huge autopkgtest suite performed by upstream on each release, and this is the reason for me using the new release as-is instead of patching old versions.

[Regression Potential]
* This is part of virtualbox on Debian and Ubuntu since bionic and 3 months

Author: Gianfranco Costamagna <email address hidden>
Date: Tue Apr 24 00:15:39 2018 +0200

basically, I now detect at runtime the --check3d switch, and enable the mesa override only for desktop and not for applications, the same as the official guest-additions packages do.

this is the code:

+ if /usr/bin/VBoxClient --check3d 2>/dev/null; then
+ if test -e /usr/lib/virtualbox/additions/00vboxvideo.conf; then
+ cp -f /usr/lib/virtualbox/additions/00vboxvideo.conf /etc/ld.so.conf.d/00vboxvideo.conf
+ ldconfig
+ fi
+ fi

+ if test -e /etc/ld.so.conf.d/00vboxvideo.conf; then
+ rm -f /etc/ld.so.conf.d/00vboxvideo.conf
+ ldconfig
+ fi

the update-alternatives switch can't work when 3d is not enabled, and with user applications using mesa-specific features.

[ upstream changes ]

Version 5.1.38 (2018-05-09)

Linux hosts: support Linux 4.17 changes. Thank you Larry Finger
Linux guests: support Linux 4.16 and EL 7.5 kernels (bugs #17676 and #17678)
Linux guests: 3D fixes for recent guests (bug #17623)

Version 5.1.36 (2018-04-17)
VMM: Fixed missing NULL pointer check in MMIO code.
Network: fixed VERR_INTNET_FLT_IF_NOT_FOUND when bridging to some adapters on Windows hosts (possibly related bugs #15294, #15060, #13481)

[Other info]
After performing an "apt-get update && apt-get upgrade" on a Ubuntu 16.04.4 box, graphics fails to start. I use my laptop with text boot, then I log in and issue "startx", with "/usr/lib/gnome-flashback/gnome-flashback-metacity" in my ~/.xinitrc file. After update, "startx" (Xorg) returns me to character prompt. Same with KDE, and while WMaker seemed to start, I still couldn't run apps like Firefox or Konsole. Tinkering with my system for about two hours I managed to figure out that this behaviour is caused by some Virtualbox package. After issuing "apt-get purge virtualbox*", the normal behaviour returned. Fortunately I don't use Virtualbox, it was installed 'just in case', so I could remove it without affecting everyday usage. But it seems that something in Virtualbox is seriously broken.

A cut from apt's history.log after upgrade and before package removal:

virtualbox:amd64 (5.0.40-dfsg-0ubuntu1.16.04.2, 5.1.34-dfsg-0ubuntu1.16.04.2)
virtualbox-dkms:amd64 (5.0.40-dfsg-0ubuntu1.16.04.2, 5.1.34-dfsg-0ubuntu1.16.04.2)
virtualbox-guest-additions-iso:amd64 (5.0.40-0ubuntu1.16.04.1, 5.1.34-0ubuntu1.16.04.2)
virtualbox-guest-dkms:amd64 (5.0.40-dfsg-0ubuntu1.16.04.2, 5.1.34-dfsg-0ubuntu1.16.04.2)
virtualbox-guest-utils:amd64 (5.0.40-dfsg-0ubuntu1.16.04.2, 5.1.34-dfsg-0ubuntu1.16.04.2)
virtualbox-guest-x11:amd64 (5.0.40-dfsg-0ubuntu1.16.04.2, 5.1.34-dfsg-0ubuntu1.16.04.2)
virtualbox-qt:amd64 (5.0.40-dfsg-0ubuntu1.16.04.2, 5.1.34-dfsg-0ubuntu1.16.04.2)

[Gianfranco Costamagna (costamagnagianfranco)]

Hello, can you please tell me if you have x11 or wayland installed?
what happens if you modprobe vboxvideo? did you try virtualbox-guest-x11-hwe instead?
please let me know and set status back to new

[Kevin Puetz (puetzk)]

I am seeing this as well. I'm actually hosting the VM in VMware workstation 12.5, but the scripts install both VMware and virtualbox guest tools (so that an OVA is useful in both).

I have an older snapshot that does not have problems, which is using 5.0.40-dfsg-0ubuntu1.16.04.2. The OVA I built today, where X crashes on startup, got 5.1.34-dfsg-0ubuntu1.16.04.2. From https://launchpad.net/ubuntu/+source/virtualbox/+publishinghistory, I don't see any versions between that built for xenial to narrow it down further.

Uninstalling virtualbox-guest-x11 makes the problem go away; X11 works again, at least in VMware. Presumably it would now not work in VirtualBox.

It seems to be happening because virtualbox-guest-x11 is stealing the alternatives for x86_64-linux-gnu_gl_conf and x86_64-linux-gnu_egl_conf, even though it's GL libraries won't work *at all* on another host. 5.0.40-dfsg-0ubuntu1.16.04.2 doesn't seem to have alternatives for these, much less high-priority defaults for them.

[Kevin Puetz (puetzk)]

Thinking about how best to solve this, maybe you move update-alternatives --install/--remove calls into a start/stop actions service triggered by detection of the appropriate environment, instead of the package postinst/prerm? I think you should be able to do it with a systemd unit that has WantedBy=graphical.target, Before=graphical.target, and ConditionVirtualization=oracle.

That (as long as those alternative links are left on --auto) should yield that VM image can just "do the right thing" based on what hardware it finds itself on, though it would still be a little messy in handling an unclean shutdown (you could start up with stale links still present). Probably not a big deal, since most reasonable use cases are authoring an image customers might launch different hosts, not actually migrating a dirty disk image around.

[Gianfranco Costamagna (costamagnagianfranco)]

Can you please remove virtualbox-guest-x11 and tell me if the problem is "fixed"? in case, yes, we should do some service to handle this.

[Kevin Puetz (puetzk)]

Yes, removing virtualbox-guest-x11 (or even just using update-alternatives to undo the redirection of for x86_64-linux-gnu_{gl,egl}_conf) restores normal function (e.g. lightdm's greeter no longer exits due to broken opengl, so the graphical login comes up and works).

[Gianfranco Costamagna (costamagnagianfranco)]

Can you please try this spin?
https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/locutusofborg-ppa/+packages

I uploaded:
 virtualbox - 5.1.34-dfsg-0ubuntu1.16.04.3
 virtualbox-hwe - 5.1.34-dfsg-0ubuntu1.16.04.3

let me know if the -x11 and -x11-hwe packages work better now :)

[Gianfranco Costamagna (costamagnagianfranco)]

ping

[Gianfranco Costamagna (costamagnagianfranco)]

I uploaded the fix in the approval queue

[Kevin Puetz (puetzk)]

Sorry for going AWOL on you. Yes, the fix in 5.1.38-dfsg-0ubuntu1.16.04.1 from https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/locutusofborg-ppa/+packages produces an OVA that works and has accelerated graphics in both VMware and VirtualBox.

One other thought in looking at how you fixed this: the notify-send "VBoxClient: the VirtualBox kernel service is not running. Exiting." is a bit distracting in VMware. I see in in looking at your updated /etc/init.d/virtualbox-guest-utils that you have an in_virtual_machine function to test if it's VirtualBox at all before probing for any kernel services. Something similar would be welcome in /etc/X11/Xsession.d/98vboxadd-xclient, perhaps:

--- /etc/X11/Xsession.d/98vboxadd-xclient
+++ 98vboxadd-xclient
@@ -15,6 +15,17 @@
 # hope that it will be useful, but WITHOUT ANY WARRANTY of any kind.
 #

+in_virtual_machine()
+{
+ if [ -z "$(lspci -d 80ee:beef)" ]; then
+ echo "VirtualBox Additions disabled, not in a Virtual Machine" >&2
+ return 1
+ fi
+
+ return 0
+}
+in_virtual_machine || return
+
 # Sanity check: if non-writeable PID-files are present in the user home
 # directory VBoxClient will fail to start.
 for i in $HOME/.vboxclient-*.pid; do

[Kevin Puetz (puetzk)]

Sorry, I have to take that back (a little): it worked in VMware and in VirtualBox, but I just realized VirtualBox hadn't set the 3d acceleration option when it imported the OVA. So your code (correctly) didn't turn on the OpenGL redirects. If I turn on the 3d acceleration, then it enables /etc/ld.so.conf.d/00vboxvideo.conf, and then graphics are broken in virtualbox (same symptoms as before, lightdm crashes).

This turned out to be because my very stripped-down image was preseeded using
   d-i base-installer/kernel/image string linux-virtual
to avoid getting lots of irrelevant drivers and firmware (stripped a few hundred megs off the OVA). But unlike linux-image-generic, linux-image-virtual doesn't depend on linux-image-extra, which means I didn't have vboxvideo.ko.

Installing virtualbox-guest-dkms fixed this by providing the missing kernel drivers. But that suggests the test for enabling /etc/ld.so.conf.d/00vboxvideo.conf isn't quite thorough enough, though this is probably a bug that should go upstream for VBoxClient --check3d instead of adding more to your script.

[Gianfranco Costamagna (costamagnagianfranco)]

the script already does the "in_virtual_machine" check
case "$1" in
  start)
        in_virtual_machine || exit 0
        log_begin_msg "Starting VirtualBox Additions"

for the check3d bug, can you please open an upstream ticket?

[Gianfranco Costamagna (costamagnagianfranco)]

@Kevin, can you please try this?
https://salsa.debian.org/pkg-virtualbox-team/virtualbox/merge_requests/1/diffs

[Łukasz Zemczak (sil2100)]

What is the status of this SRU? Are the versions in the xenial Unapproved queue good to go in the current shape?

[Gianfranco Costamagna (costamagnagianfranco)]

I would say so, nobody reported regressions so far.

[Kevin Puetz (puetzk)]

I was suggesting that /etc/X11/Xsession.d/98vboxadd-xclient should do the same in_virtual_machine check, which I indeed adapted from what you currently do in /etc/init.d/virtualbox-guest-utils

[Kevin Puetz (puetzk)]

And yes, my second problem vboxvideo.ko is not a regression from the status quo; the old package *always* turns on the libGL redirection, which breaks everybody that's not virtualbox with 3d accel enabled and working. The proposed fix only turns it on if VBoxClient --check3d returns success, which avoids conflicting with VMware, real PCs, VirtualBox with 3d acceleration disabled, etc.

There's still an issue if you're on virtualbox, have 3d acceleration turned on, but are missing the kernel driver for it to actually work. In that case VBoxClient --check3d succeeds, the libGL redirection is enabled, but it doesn't actually work. But anyone in that situation is also broken with today's unconditionally-enabled package (and has a proper fix; either turn off the configuration for 3d accel if they didn't intend to use it, or install the kernel driver using virtualbox-guest-dkms so that it does work).

So the current package in the queue seems good to merge as-is. It fixes most broken configurations, doesn't break any that were currently working, and the remaining problem seems like an independent upstream bug.

[Gianfranco Costamagna (costamagnagianfranco)]

>I was suggesting that /etc/X11/Xsession.d/98vboxadd-xclient should do the same in_virtual_machine check, which I indeed adapted from what you currently do in /etc/init.d/virtualbox-guest-utils

feel free to provide a minimal/tested patch for this!

[Raffaele Forte (raffaele.forte)]

Tested and working for me!

[Kevin Puetz (puetzk)]

> feel free to provide a minimal/tested patch for this!

I did, in the comment where I first mentioned it: https://bugs.launchpad.net/ubuntu/+source/virtualbox/+bug/1762491/comments/9

I'm applying that patch patching in my current builds, and the resulting OVA has working Xsession guest additions in virtualbox (e.g. seamless mode, clipboard) and no spurious warnings in VMware.

[Łukasz Zemczak (sil2100)]

Ok, basing on comments #14 and #16, I will be reviewing what's currently in the Unapproved queue. Thanks!

[Łukasz Zemczak (sil2100)]

Hello Akos, or anyone else affected,

Accepted virtualbox into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/virtualbox/5.1.38-dfsg-0ubuntu1.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Akos, or anyone else affected,

Accepted virtualbox-guest-additions-iso into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/virtualbox-guest-additions-iso/5.1.38-0ubuntu1.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Akos, or anyone else affected,

Accepted virtualbox-ext-pack into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/virtualbox-ext-pack/5.1.38-0ubuntu1.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Akos, or anyone else affected,

Accepted virtualbox-hwe into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/virtualbox-hwe/5.1.38-dfsg-0ubuntu1.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Gianfranco Costamagna (costamagnagianfranco)]

can anybody please give it a test?

[Gianfranco Costamagna (costamagnagianfranco)]

@Kevin, your fix is now in cosmic!

[Gianfranco Costamagna (costamagnagianfranco)]

@Kevin, can you please tell me if cosmic has still the issues you mentioned above? e.g. the check3d switch might work correctly with vbox 5.2+

[Akos Nagy (nagyak)]

For me the bug is fixed with version 5.1.38-dfsg-0ubuntu1.16.0. Thanks to all of you for working on it!

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for virtualbox has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package virtualbox - 5.1.38-dfsg-0ubuntu1.16.04.1

---------------
virtualbox (5.1.38-dfsg-0ubuntu1.16.04.1) xenial; urgency=medium

  * New upstream version 5.1.38-dfsg
    - fixes for kernel 3.16 and 3.17
    - fixes for opengl libraries
    - automatically detect if 3d is enabled or not
    LP: #1762491

 -- Gianfranco Costamagna <email address hidden> Tue, 10 Jul 2018 12:40:22 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package virtualbox-ext-pack - 5.1.38-0ubuntu1.16.04.1

---------------
virtualbox-ext-pack (5.1.38-0ubuntu1.16.04.1) xenial; urgency=medium

  * New upstream version 5.1.38 (LP: #1762491)
  * New upstream release, update rules, postinst file
  * Update copyright file
  * Update watch file
  * Automatically generate postinst version

 -- Gianfranco Costamagna <email address hidden> Tue, 10 Jul 2018 12:49:47 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package virtualbox-guest-additions-iso - 5.1.38-0ubuntu1.16.04.1

---------------
virtualbox-guest-additions-iso (5.1.38-0ubuntu1.16.04.1) xenial; urgency=medium

  * New upstream version 5.1.38
    - LP: #1762491
  * Update copyright file

 -- Gianfranco Costamagna <email address hidden> Tue, 10 Jul 2018 13:04:33 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package virtualbox-hwe - 5.1.38-dfsg-0ubuntu1.16.04.1

---------------
virtualbox-hwe (5.1.38-dfsg-0ubuntu1.16.04.1) xenial; urgency=medium

  * Build only the guest-* packages with hwe stack

virtualbox (5.1.38-dfsg-0ubuntu1.16.04.1) xenial; urgency=medium

  * New upstream version 5.1.38-dfsg
    - fixes for kernel 3.16 and 3.17
    - fixes for opengl libraries
    - automatically detect if 3d is enabled or not
    LP: #1762491

 -- Gianfranco Costamagna <email address hidden> Mon, 09 Jul 2018 17:10:10 +0200