calculation of needed free space in /boot is inaccurate and causes refusal to upgrade

[Steve Langasek]

[SRU Justification]
In some configurations, update-manager will refuse to upgrade kernel packages, citing lack of space in /boot even though there is enough space in /boot and if the upgrade is done with 'apt dist-upgrade', it succeeds. Affected users may as a result fail to have security updates applied to their system, unless they know how to run apt from the commandline or have unattended upgrades enabled.

[Test case]
1. Check the space currently used on /boot with du -sh.
2. Create a partition 120MB larger than the space used on /boot.
3. Copy the contents of your /boot directory into this partition.
4. Mount this partition as /boot.
5. Configure your system so that a newer kernel ABI is available (e.g., by enabling -proposed; or by waiting for the next kernel update to publish; or by downgrading the kernel using apt-get and rebooting).
6. Run update-manager.
7. Verify that update-manager refuses to upgrade, citing a lack of space in /boot.
8. Install python3-distupgrade and ubuntu-release-upgrader-core from -proposed.
9. Re-run update-manager.
10. Verify that update-manager proceeds with the upgrade, and that the upgrade succeeds, with no failure due to lack of disk space on /boot.
11. Repeat steps 2-9 with a partition that is only 60MB larger than the space used on /boot.
12. Verify that update-manager still refuses to proceed with the upgrade, with the packages from -proposed.

[Regression potential]
Since this changes the limit at which update-manager refuses to upgrade due to lack of free space, if the new calculation of the minimum required free space is too low, users may see update-manager proceed with the upgrade, then fail at the dpkg stage, requiring manual recovery.

[Original report]
I have a /boot partition which is sized just right to accommodate 3 kernels, plus a little bit of overhead.

$ df -h /boot/
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 280M 144M 118M 56% /boot
$

Previously, update-manager had no problem letting me upgrade kernels. With the latest kernel update in 16.10, I got an error message that I needed more free space on this filesystem.

After bypassing update-manager with 'sudo apt dist-upgrade', the filesystem's usage looks like this:

$ df -h /boot/
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 280M 206M 56M 79% /boot
$

So update-manager is getting the disk usage requirement for /boot off by a factor of 2 (at least? I didn't save the error message from the dialog).

It's better to refuse to upgrade than to fail mid-upgrade due to disk space, but not if that means getting stuck forever with an old kernel unnecessarily.

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: update-manager 1:16.10.7
ProcVersionSignature: Ubuntu 4.8.0-27.29-generic 4.8.1
Uname: Linux 4.8.0-27-generic x86_64
ApportVersion: 2.20.3-0ubuntu8
Architecture: amd64
CurrentDesktop: Unity
Date: Wed Nov 30 12:10:25 2016
GsettingsChanges:
 b'com.ubuntu.update-manager' b'show-details' b'true'
 b'com.ubuntu.update-manager' b'window-height' b'551'
 b'com.ubuntu.update-manager' b'first-run' b'false'
 b'com.ubuntu.update-manager' b'window-width' b'813'
 b'com.ubuntu.update-manager' b'launch-time' b'1480522730'
InstallationDate: Installed on 2010-09-24 (2259 days ago)
InstallationMedia: Ubuntu 10.04.1 LTS "Lucid Lynx" - Release amd64 (20100816.1)
PackageArchitecture: all
SourcePackage: update-manager
UpgradeStatus: Upgraded to yakkety on 2016-10-28 (32 days ago)

[Brian Murray (brian-murray)]

I think part of the issue here is the function estimate_kernel_size_in_boot, from utils.py, which does the following:

 92 def estimate_kernel_size_in_boot():
 93 """ estimate the amount of space that the current kernel takes in /boot """
 94 size = 0
 95 kver = os.uname()[2]
 96 for f in glob.glob("/boot/*%s*" % kver):
 97 size += os.path.getsize(f)
 98 return size

The glob ends up including vmlinuz and initrd, which is fine but then DistUpgradeCache.py multiples size by 2. We actually don't need enough free space to accomadate for two of every kernel file in boot though.

[Steve Langasek (vorlon)]

Related bug, which led to the current behavior: bug #1173468

At the time, the decision was made to double the calculated value for required free space. This is better than being undersized, but it's over large; this sets the min free space value to 2 * N_kernels * (sizeof(kernel_package_bits) + sizeof(initramfs)).

The high water mark for disk usage is actually:

 1) all the newly installed kernels are unpacked to disk
 2) any kernels being upgraded are unpacked onto disk, but have not yet replaced the old files - i.e., unpacked as .dpkg-new (technically, I think this can only be true of one upgraded kernel package at a time, but it's highly unusual to ever have *a* kernel package being upgraded and exponentially unusual to have more than one)
 3) an initrd on disk for each of the new and upgraded kernels
 4) one additional initrd on disk (initrd.img-$version.new) being prepared as a replacement.

It's possible that 2) and 4) would never be unpacked at the same time, in which case the high water mark would actually be 1 + 3 + MAX(2,4). But this is only slightly smaller than 1+2+3+4 so probably doesn't matter in practice.

I think therefore that we should calculate N_kernels * sizeof(kernel_package_bits) + (N_kernels + 1) * sizeof(initramfs) and use that as the freespace value. This will require a moderate refactor between update-manager and ubuntu-release-upgrader.

The current fudge factor for kernel growth over time - currently, 1MiB additional per kernel - should be retained.

[Steve Langasek (vorlon)]

Output of update-manager --debug, per request:

$ update-manager --debug

(update-manager:14909): dconf-WARNING **: failed to commit changes to dconf: GDBus.Error:org.gtk.GDBus.UnmappedGError.Quark._g_2dfile_2derror_2dquark.Code12: Failed to write file '/home/vorlon/.config/dconf/user.YF08RY': write() failed: No space left on device
DEBUG:AptClient.DebconfProxy:debconf socket: /tmp/aptdaemon-0_qlhknq/debconf.socket
DEBUG:root:debconf.start()
DEBUG:root:debconf.stop()
INFO:root:holding back phased update libsnapd-glib1 (0 < 53)
INFO:root:holding back phased update snapd-login-service (0 < 53)
DEBUG:root:App candidate for mythtv-frontend: /usr/share/app-install/desktop/mythtv-frontend:mythtv.desktop
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:Useless call to _add_deps. Package was None or didn't have a candidate.
DEBUG:root:WARNING: can not get name for '<Gtk.TreeViewAccessible object at 0x7f969520b828 (GtkTreeViewAccessible at 0x55bc8851f300)>'
DEBUG:root:WARNING: can not get name for '<Gtk.LabelAccessible object at 0x7f969520f9d8 (GtkLabelAccessible at 0x55bc88c0e6c0)>'
DEBUG:root:WARNING: can not get name for '<Gtk.TreeSelection object at 0x7f969520fe58 (GtkTreeSelection at 0x55bc88dce080)>'
DEBUG:root:WARNING: can not get name for '<Gtk.TextViewAccessible object at 0x7f969519c240 (GtkTextViewAccessible at 0x55bc88bd4730)>'

(update-manager:14909): Gtk-WARNING **: Could not get code point of keysym \

(update-manager:14909): Gtk-WARNING **: Could not get code point of keysym )

(update-manager:14909): Gtk-WARNING **: compose file /home/vorlon/.XCompose does not include any keys besides keys in en-us compose file

(update-manager:14909): Gtk-WARNING **: Could not get code point of keysym \

(update-manager:14909): Gtk-WARNING **: Could not get code point of keysym )

(update-manager:14909): ...

[Steve Langasek (vorlon)]

The update-manager estimate_kernel_size_in_boot() function contains the current bug, but this function also has no consumers anywhere except in ubuntu-release-upgrader. So I'm just going to move this code into u-r-u to avoid having to update both packages.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:17.04.6

---------------
ubuntu-release-upgrader (1:17.04.6) zesty; urgency=medium

  * DistUpgrade/DistUpgradeCache.py: import kernel initrd size estimation
    code from update-manager (since u-r-u is the only consumer of it) and
    refactor in order to give more accurate estimates. LP: #1646222.

 -- Steve Langasek <email address hidden> Wed, 21 Dec 2016 15:41:25 -0600

[Steve Langasek (vorlon)]

This is fixed in zesty, but somehow the bug did not get closed when it migrated.

ubuntu-release-upgrader (1:17.04.6) zesty; urgency=medium

  * DistUpgrade/DistUpgradeCache.py: import kernel initrd size estimation
    code from update-manager (since u-r-u is the only consumer of it) and
    refactor in order to give more accurate estimates. LP: #1646222.

 -- Steve Langasek <email address hidden> Wed, 21 Dec 2016 15:41:25 -0600

[Łukasz Zemczak (sil2100)]

Hello Steve, or anyone else affected,

Accepted ubuntu-release-upgrader into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:16.04.21 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Steve Langasek (vorlon)]

we should eventually drop the code from update-manager, but it doesn't need to be done in SRU.

[Robie Basak (racb)]

In reviewing 1:16.10.10 in the Yakkety unapproved queue, I see changes to data/mirrors.cfg that aren't explained in the changelog or in this bug. They may be sorting noise, but SRUs are supposed to be minimal. So is this intentional?

[Steve Langasek (vorlon)]

On Wed, Jan 25, 2017 at 03:52:06PM -0000, Robie Basak wrote:
> In reviewing 1:16.10.10 in the Yakkety unapproved queue, I see changes
> to data/mirrors.cfg that aren't explained in the changelog or in this
> bug. They may be sorting noise, but SRUs are supposed to be minimal. So
> is this intentional?

That is a file that's autogenerated and updated at source package build time
based on the current list of published mirrors. This change is correct.

[Brian Murray (brian-murray)]

I've verified this fix for Xenial with the version of the package in -proposed but it might be worth noting that I first verified the refusal to upgrade with the package from -proposed by creating a 40M file on my /boot partition. I then removed the 40M file and upgraded rather than repeating steps 2 through 9.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:16.04.21

---------------
ubuntu-release-upgrader (1:16.04.21) xenial; urgency=medium

  * DistUpgrade/DistUpgradeCache.py: import kernel initrd size estimation
    code from update-manager (since u-r-u is the only consumer of it) and
    refactor in order to give more accurate estimates. LP: #1646222.

 -- Steve Langasek <email address hidden> Thu, 22 Dec 2016 00:05:53 -0600

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Brian Murray (brian-murray)]

Hello Steve, or anyone else affected,

Accepted ubuntu-release-upgrader into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:16.10.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Yakkety only need 111M free so I had to fill up boot some more to generate the error message about needing to free up space in /boot. I then installed the packages from yakkety-proposed and verified I still saw an error with a fuller disk and then that the upgrade proceeded with 102M (less than 111M and more than 99M) free space in /boot.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-release-upgrader - 1:16.10.10

---------------
ubuntu-release-upgrader (1:16.10.10) yakkety; urgency=medium

  * DistUpgrade/DistUpgradeCache.py: import kernel initrd size estimation
    code from update-manager (since u-r-u is the only consumer of it) and
    refactor in order to give more accurate estimates. LP: #1646222.

 -- Steve Langasek <email address hidden> Wed, 21 Dec 2016 17:54:07 -0600

[Launchpad Janitor (janitor)]

This bug was fixed in the package update-manager - 1:17.04.2

---------------
update-manager (1:17.04.2) zesty; urgency=medium

  [ Jeremy Bicha ]
  * Update alternate dependencies for policykit-1-gnome:
    - Add virtual polkit-1-auth-agent
    - Replace non-existant policykit-1-kde with polkit-kde-agent-1
    - Replace razorqt-policykit-agent with lxqt-policykit
      (since lxqt is the replacement for razorqt)

  [ Manzur Mukhitdinov ]
  * UpdateManager/Core/MetaRelease.py:
    - Add a clearer error message when a network failure occurs

  [ Brian Murray ]
  * UpdateManager/Dialogs.py:
    - Remove misleading ellipsis in “Restart Now” action button. Thanks to
      Adolfo Jayme for the change. (LP: #1568368)

  [ Steve Langasek ]
  * Drop estimate_kernel_size_in_boot() from UpdateManager/Core/utils.py; this
    was only used in ubuntu-release-upgrader so the code has now moved there
    (with code fixes). Ensure upgrade ordering with Breaks: on older
    python3-distupgrade. LP: #1646222.

 -- Steve Langasek <email address hidden> Wed, 08 Mar 2017 10:01:30 -0800

[Jarno Suni (jarnos)]

I do not understand that why the code is used in ubuntu-release-upgrader only. Doesn't update-manager need to check free space before installing new kernels? At least some version of updater-manager may display a dialog like this https://i.stack.imgur.com/DppOi.png ?

[Jarno Suni (jarnos)]

If patch https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1678187/comments/8 is applied, there is no need for one extra sizeof(initramfs) in separate /boot since the new file is created in "${TMPDIR:-/var/tmp}" and the old one is only replaced by it in /boot. But still some other partition may be too full for kernel update. E.g. /usr/src may run out of inodes due to linux-headers-* packages.

[Steve Langasek (vorlon)]

On Sat, Aug 05, 2017 at 06:32:47AM -0000, Jarno Suni wrote:
> I do not understand that why the code is used in ubuntu-release-upgrader
> only.

Because update-manager depends on python3-distupgrade and imports the size
checking code from there. It does not make sense to have two copies of this
implementation (with different bugs).

> Doesn't update-manager need to check free space before installing
> new kernels? At least some version of updater-manager may display a
> dialog like this https://i.stack.imgur.com/DppOi.png ?

Yes, and this now uses the code in ubuntu-release-upgrader instead of having
update-manager call into u-r-u's python module and u-r-u call back into
update-manager.

On Sat, Aug 05, 2017 at 07:09:28AM -0000, Jarno Suni wrote:
> If patch https://bugs.launchpad.net/ubuntu/+source/initramfs-
> tools/+bug/1678187/comments/8 is applied, there is no need for one extra
> sizeof(initramfs) in separate /boot since the new file is created in
> "${TMPDIR:-/var/tmp}" and the old one is only replaced by it in /boot.
> But still some other partition may be too full for kernel update. E.g.
> /usr/src may run out of inodes due to linux-headers-* packages.

Creating the file in /var/tmp means that it cannot atomically replace the
existing initramfs; a mv from /var/tmp to /boot then becomes a non-atomic
removal and replace, which means there is a window during which the
initramfs on disk for the default kernel will be broken and a power event
will cause the system to fail to reboot. That is not a trade-off we are
ever going to make.

The right fix for the out-of-space-on-removal problem is to fix the kernel
packaging to not regenerate initramfs on removals of linux-image-extras or
linux-image-signed.