The disk-space-needed estimate doesn't account for the snaps

[Sebastien Bacher]

[Impact]

Since there have been reports that users might fail to boot to a working gdm in the case of a full disk, we need to make sure users upgrading from and older release to disco/eoan have enough space for the upgrade to succeed. Currently we are installing snaps to replace certain debs on the system (+ refreshing existing snaps to newer channels/branches). The additional size required for the snap replacements was not taken into consideration when performing the upgrade, which might lead to edge-cases of users not having enough space for the upgrade to succeed.

[Test Case]

* Prepare a bionic machine
* Make sure ubuntu-desktop and snapd are installed on it
* Install core18 from stable and, for instance, gnome-characters from the stable/ubuntu-18.04 branch
* Do snap list and snap refresh other gnome-related snaps to stable/ubuntu-18.04
* Perform an upgrade to disco (with the --proposed option, to use disco-proposed)
* Reboot
* Make sure the update succeeded and all the installed gnome snaps now track stable/ubuntu-19.04 instead
* Check /var/log/dist-upgrade/main.log and see if '/var' is listed in the calculated required space print-out

Additional test case to make sure update-manager did not regress due to the changes:

* On the newly upgraded system (or a new disco instance with ubuntu-release-upgrader upgraded to the version in disco-proposed), downgrade any of the installed packages
* On the terminal, run `update-manager` and wait for the window to appear
* Once the list of available upgrades appears, press the 'install now' button
* Upgrades should install and no python traceback should be printed on the terminal

[Regression Potential]

As with any change touching the upgrade code-path, this change also carries a risk and should be well tested before going forward. That being said, the fix has been specifically basically limited to DistUpgradeQuirks to make regressions less likely. Additionally, unit tests have been added for every modified code-path.
If any regressions should be found, the most likely place for those would be in size estimation, causing users to get notified of lack of space when they shouldn't. Another possible location would be all the things regarding replacing the debs with snaps - so either failures in snap replacement installation or refreshing.

[Original Description]

Looking at the code change recently made to install snaps on upgrade, it seems like there was no change made to the disk-space-estimate code. We had some reports recently that bionic might fail to boot to a working gdm on full disk so that could potentially be an important issue for upgraders

[Brian Murray (brian-murray)]

Given that the snaps are located in /var/lib/snapd and we currently allocate free space to download debs in /var/cache/apt/archives and that it is a corner case for those to not be the same mount point it is likely that the space allocated for /var/cache/apt/archives would be enough free space. However, the release upgrader doesn't currently clean up the download .debs at all let alone before installing the snaps. So one solution would be to have the release upgrader clean up after itself.

[Brian Murray (brian-murray)]

As an interim solution I've modified the release-upgrader to clean up the downloaded debs before installing the snaps.

https://bazaar.launchpad.net/~ubuntu-core-dev/ubuntu-release-upgrader/trunk/revision/3153

[Mathieu Trudel-Lapierre (cyphermox)]

Maybe we want to also revise the minimal disk space calculations for installers, in case there's a bit more space needed for preinstalled snaps?

[Julian Andres Klode (juliank)]

FWIW, same for APT and any kind of installer debs

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:19.10.11

---------------
ubuntu-release-upgrader (1:19.10.11) eoan; urgency=medium

  * DistUpgrade/DistUpgradeQuirks.py, DistUpgrade/DistUpgradeCache.py:
    Modify DistUpgradeQuirks to calculate extra space needed for snaps that
    need to be installed during upgrade (replacing old debs) and add that to
    estimates done in checkFreeSpace(). (LP: #1783597)
  * DistUpgrade/DistUpgradeQuirks.py: Make sure snaps like core18 are not
    refreshed/installed from the stable/ubuntu-VERSION branch but instead use
    stable on upgrades. (LP: #1841225)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 03 Sep 2019 15:58:38 +0200

[Kiwinote (kiwinote)]

Hello. Just so that the relevant people see it - this upload would appear to cause the install button in update-manager to stop working as per https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1843076

[Łukasz Zemczak (sil2100)]

Just leaving a note here for the purposes of the SRU: the regression mentioned in #6 has been fixed and the package in disco Unapproved modified to include the fix. The current package in the queue should be good for review. I'll modify the test case to make sure we check that update-manager is still working after upgrade.

[Brian Murray (brian-murray)]

Hello Sebastien, or anyone else affected,

Accepted ubuntu-release-upgrader into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:19.04.16.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of ubuntu-release-upgrader from disco-proposed was performed and bug 1847313 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed-disco" tag from this bug report and add the tag "bot-stop-nagging" to bug 1847313 (not this bug). Thanks!

[Łukasz Zemczak (sil2100)]

The above bug does not seem to be related to any of the changes provided by the SRU. The mainlog there mentions:
2019-10-08 14:12:33,569 ERROR No 'ubuntu-minimal' available/downloadable after sources.list rewrite+update
2019-10-08 14:12:33,569 ERROR 'ubuntu-minimal' was not downloadable

Anyway, I will double-check to make sure all is good and switch the tags after I perform my validation per the test-case.

[Łukasz Zemczak (sil2100)]

Verified ubuntu-release-upgrader 19.04.16.7 as per the test-case.

I set up a multipass bionic system, switched /etc/update-manager/release-upgrades to Prompt=normal, installed ubuntu-desktop, installed the core18 snap (from stable) and gnome-characters snap (from stable/ubuntu-18.04). I then proceeded with running `do-release-upgrade --frontend DistUpgradeView --proposed`. The upgrade succeeded, I have confirmed that 19.04.16.7 was used [1], checked that all snaps have been refreshed correctly. Also, checked that '/var' is listed in the snap estimates [2].

Will still check if update-manager is fine before switching the tag to verification-done.

[1] 2019-10-09 13:28:16,936 INFO release-upgrader version '19.04.16.7' started
[2] 2019-10-09 13:29:05,882 DEBUG dir '/var' needs '9207808' of '<DistUpgrade.DistUpgradeCache.MyCache.checkFreeSpace.<locals>.FreeSpace object at 0x7f837003a278>' (9367467576.000000)

[Łukasz Zemczak (sil2100)]

Ok, so I used my own system to verify that update-manager is still working. I have upgraded ubuntu-release-upgrader packages to the version in disco-proposed and ran update-manager. I was able to successfully install all upgrades.

ubuntu-release-upgrader-core:
  Installed: 1:19.04.16.7
  Candidate: 1:19.04.16.7
  Version table:
 *** 1:19.04.16.7 100
        100 /var/lib/dpkg/status

Considering the bug as verified.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:19.04.16.7

---------------
ubuntu-release-upgrader (1:19.04.16.7) disco; urgency=medium

  * DistUpgrade/DistUpgradeQuirks.py, DistUpgrade/DistUpgradeCache.py:
    Modify DistUpgradeQuirks to calculate extra space needed for snaps that
    need to be installed during upgrade (replacing old debs) and add that to
    estimates done in checkFreeSpace(). (LP: #1783597)
  * DistUpgrade/DistUpgradeCache.py: Make sure checkFreeSpace() still works for
    update-manager with its own implementation of MyCache.
  * DistUpgrade/DistUpgradeQuirks.py: Make sure snaps like core18 are not
    refreshed/installed from the stable/ubuntu-VERSION branch but instead use
    stable on upgrades. (LP: #1841225)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Mon, 09 Sep 2019 15:41:12 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.