Ubuntu needs a QA test case for HWE EOL upgrades

[Erick Brunzell]

Presumably sometime in August those who installed Trusty using the 14.04.2, 14.04.3, and 14.04.4 media will be presented with HWE EOL warnings, and they'll have the choice of either upgrading to Xenial or performing an HWE upgrade. No test case exists for HWE upgrades.

I began saving a list of web links of problems encountered when Precise users were notified of HWE EOL in anticipation of recommending that we amend our download page to recommend users always try the first point release. I chose not to do that because the 14.04.1 images were affected by bug #1265192 which could result in data loss. (That may be worth pursuing prior to 16.04.2 but that's a whole different issue).

Anyway here's a partial list of problems encountered when Precise users hit HWE EOL (I'd think we'd want to avoid this for Trusty HWE EOL):

http://ubuntuforums.org/showthread.php?t=2234260 (dependencies)

http://ubuntuforums.org/showthread.php?t=2234815 (dependencies)

http://ubuntuforums.org/showthread.php?t=2234813 (dependencies)

http://ubuntuforums.org/showthread.php?t=2234830 (free space in /boot)

http://ubuntuforums.org/showthread.php?t=2234693 (dependencies)

http://ubuntuforums.org/showthread.php?t=2234922 (no desktop - kernel update w/o Xorg?)

http://ubuntuforums.org/showthread.php?t=2234729 (dependencies)

http://ubuntuforums.org/showthread.php?t=2235159 (dependencies)

http://ubuntuforums.org/showthread.php?t=2235287 (who knows?)

http://ubuntuforums.org/showthread.php?t=2235698 (partial upgrade, not HWE related)

http://ubuntuforums.org/showthread.php?t=2236081 (dependencies)

http://ubuntuforums.org/showthread.php?t=2235843 (insufficient info)

http://askubuntu.com/questions/493541/hardware-enablement-stack-hwe-out-of-support (dependencies)

http://ubuntuforums.org/showthread.php?t=2236096 (insufficient info)

http://ubuntuforums.org/showthread.php?t=2235113 (might have been a dist-upgrade)

http://ubuntuforums.org/showthread.php?t=2236129 (insufficient info)

https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1328264 (dependencies bug)

http://ubuntuforums.org/showthread.php?t=2236584 (low res w/ nvidia)

http://ubuntuforums.org/showthread.php?t=2237132 (insufficient info)

http://ubuntuforums.org/showthread.php?t=2237277 (insufficient info)

[Kev Bowring (flocculant)]

Write one. https://wiki.ubuntu.com/QATeam/ContributingTestcases/Manual/Writing

Easy enough to then add it to the tracker testcases.

Once that's done then those who want to use it can add it to their tests.

[Erick Brunzell (lbsolost)]

I'd have no idea how to write a test case for this. I was just looking at the Precise HWE EOL wiki and one of the prime bits would be waiting for the addition of 'hwe-support-status' to 'update-manager-core'. Or maybe we'd have to rely on a version still in bzr?????

This is complex enough it would at the very least require involvement of the Xorg team and Kernel team (since those are the main components upgraded in an HWE EOL upgrade), but also maintainers of 'update-manager-core'.

The Trusty dailies are still on the lts-wily HWE bits but once the lts-xenial bits appear to have all landed I thought about trying a manual HWE upgrade similar to this (I'd just replace wily with xenial):

https://wiki.ubuntu.com/Kernel/LTSEnablementStack#Desktop

Of course an apt-cache policy search for each package may be wise to rule out different sonames and such, but that still may not be truly representative of what the update-manager does via hwe-support-status.

All in all it requires someone much smarter than I am. I'm just looking for some way of avoiding what was a very buggy Precise HWE EOL experience when Trusty reaches that point in the coming weeks.

[Brian Murray (brian-murray)]

I added the HWE support information to update-manager for Trusty (bug 1498059) so can provide you with information as to what a manual test case would involve. I've just approved (literally minutes ago) a new version of update-manager in trusty-proposed which fixes a couple of strings namely recommending to upgrade from 12.04 to 14.04.

There are two parts to the HWE support in update-manager. One is the tool hwe-support-status which provides information about the support for your HWE stack. After 8/4/2016, when support ends for the U,V,W HWE stacks, it will provide information about how to upgrade. Some tests for hwe-support-status would include:

Run hwe-support-status and observe it security updates are no longer available for your HWE stack.
  * If update-manager is installed it will recommend upgrading via update-manager.
  * If update-manager is not installed it will provide a list of packages to upgrade via apt-get.

Running hwe-support-status with --show-replacements should list packages from the next LTS e.g. xenial.

Running hwe-support-status with --show-unsupported should list your current HWE stack packages e.g. with lts-vivid in them.

Running hwe-support-status --quiet should exit with code 10.

[Brian Murray (brian-murray)]

The update-manager gui will also behave differently if your HWE stack is no longer supported. If your system is out of date (e.g. there are some packages in -updates not installed) then you will be presented with a dialog regarding "New and important security and hardware support update". The dialog will have an install button which will upgrade your HWE stack to the next LTS. After installing the updates and rebooting you can run hwe-support-status as described in comment #3 and should not see anything unsupported.

update-manager will recommend release upgrades over installing a new HWE stack so if all your packages are up to date you will not see the "New and important security..." dialog. One way to force the dialog is to set /etc/update-manager/release-upgrades "Prompt=never".

[Steve Langasek (vorlon)]

Trying to filter through this *very* long list of links:

> http://ubuntuforums.org/showthread.php?t=2234260
> http://ubuntuforums.org/showthread.php?t=2234813
> http://ubuntuforums.org/showthread.php?t=2234729
> http://ubuntuforums.org/showthread.php?t=2235159
> http://ubuntuforums.org/showthread.php?t=2236081

These all show an error message that makes no sense ("Depends: $pkg (= $version) but $version is to be installed"). Doesn't seem related to hwe eol in any way. Must be a bug in the package manager in 12.04, which is unlikely to recur in the same form and so doesn't warrant a test for this specifically.

> http://ubuntuforums.org/showthread.php?t=2234815

This shows an error trying to install the trusty hwe stack on a system that had the quantal stack installed. I would argue this should be an automated test rather than a manual one...

> http://ubuntuforums.org/showthread.php?t=2234830

Possibly video-hardware-specific; it is not guaranteed that all hardware will be supported after switching HWE stacks.

> http://ubuntuforums.org/showthread.php?t=2234693

This shows the 'apt-get' command included in the hwe-support-status output doesn't work. Should also be an automated test.

> http://ubuntuforums.org/showthread.php?t=2234922

User reinstalled without root-causing the failure, so not testable.

> http://ubuntuforums.org/showthread.php?t=2235287

"can't log in after update" - not directly related to the hwe package upgrades; not testable.

> http://ubuntuforums.org/showthread.php?t=2235698

ubuntu-desktop reported removed. Yes this should be tested (although the submitter refers to a "partial upgrade", which is not what should happen normally).

> http://ubuntuforums.org/showthread.php?t=2235843

Likely another case of packages being removed incorrectly on upgrade; without more details, not really actionable.

> http://askubuntu.com/questions/493541/hardware-enablement-stack-hwe-out-of-support
> https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1328264

This is interesting, it suggests our upgrade instructions may need modification for the multiarch case...

> http://ubuntuforums.org/showthread.php?t=2236096

It is not guaranteed that all hardware will be supported after switching HWE stacks.

> http://ubuntuforums.org/showthread.php?t=2235113

Unclear to me that this is related to HWE EOL; if it is, probably a duplicate of another thread by the same poster.

> http://ubuntuforums.org/showthread.php?t=2236129

Complaint about specific hardware not functioning the same with drivers in the new kernel. Not testable, not fixable in the general case.

> http://ubuntuforums.org/showthread.php?t=2236584

Says the nvidia driver was removed on upgrade. Should be testable.

> http://ubuntuforums.org/showthread.php?t=2237132

Insufficient information to know what this was; probably covered by one of the above cases.

> http://ubuntuforums.org/showthread.php?t=2237277

Probably a repeat of the nvidia driver problem.

[Brian Murray (brian-murray)]

The multi-arch issue Steve mentions looks like bug 1349400 and might also be a good thing to test in Trusty.

[Erick Brunzell (lbsolost)]

I had done some testing on a Trusty w/Utopic HWE install yesterday and updated update-manager from trusty-proposed. I booted that install this afternoon and was notified of the HWE EOL upgrade. After checking that I'd disabled the proposed repos I looked and saw that "Notify of new release" was set to LTS only, but I'm sure after being notified of the release-upgrade yesterday I declined.

So with everything seemingly in order I accepted the HWE upgrade along with some others (including Firefox) and all seemed to complete without a problem. I was prompted to reboot but upon doing so the screen freezes with "Restoring resolver state" regardless of which kernel I attempt to boot so I filed bug 1610434 (poorly no doubt).

[Erick Brunzell (lbsolost)]

I cobbled together a test process and found that the cause for bug #1610434 is that the HWE EOL upgrade process is failing to install libwayland-egl1-mesa-lts-xenial. I did perform one test each on Ubuntu and Lubuntu which seemed to work just fine.