Kernels not autoremoving, causing out of space error on LVM or Encrypted installation or on any installation, when /boot partition gets full

[Elfy]

Currently if one chooses to use LVM or encrypted install, a /boot partition is created of 236Mb

Once kernel updates start being released this partition soon fills until people are left unable to upgrade.

While you and I might know that we need to watch partition space, many of the people we have installing think that a windows disk is a disk and not a partition, education is probably the key - but in the meantime support venues keep needing to deal with the fact the partition is too small and/or old kernels are not purged as new ones install.

For workaround and sytem repair, see
https://help.ubuntu.com/community/RemoveOldKernels

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Elfy (elfy)]

Installed with LVM in a vm, /boot size 236M

/dev/sda1 236M 38M 186M 17% /boot

http://pastebin.ubuntu.com/8062443/

Installing with encryption and LVM results in the same /boot size unsurprisingly

[varunendra (varunendra)]

Just dealt with the problem of an affected user here : http://ubuntuforums.org/showthread.php?t=2240240

But I guess adding myself to the "Affects Me" list would be cheating? Since it is not me who is affected.

[TJ (tj)]

The issue here isn't so much the absolute space initially allocated to the /boot/ LV, it is that the installer allocates 100% of the Volume Groups space.

The great advantage of LVM is that space can by dynamically allocated and removed from Logical Volumes as needed.

If the installer were to leave 5% of the Volume Group unallocated it would be possible for the system to detect this out-of-space condition and extend the /boot/ LV:

lvextend -l 50%FREE VG/boot
resize2fs /dev/mapper/VG-boot # assuming ext file-system

and at the same time display a prominent warning and options for automatically removing the superseded kernel versions.

[Rafael Gattringer (rafael.gattringer)]

Especially a problem when setting up systems for clients, who you only visit once or twice a year. The kernel updates fill the boot partition fast and so users are warned at the start that no space is left.

[Donald Tripp (dtlongrifles)]

I have this problem with Xubuntu 14.04

[Sergey Romanov (sml-uni)]

Link to yet another forum thread: http://ubuntuforums.org/showthread.php?t=2256479
The problem is twofold.

1) Users not paying attention to apt-get saying:

The following packages were automatically installed and are no longer required:
...
Use 'apt-get autoremove' to remove them.

like in the thread linked above.

2) Systems where linux-image-<version>-{generic,server,goldfish,lowlatency} don't get suggested for autoremoval after upgrade to newer versions of linux-{generic,server,goldfish,lowlatency}/linux-image-{generic,server,goldfish,lowlatency}. Mostly because some dkms-based kernel-modules are installed like NVIDIA/AMD propietary video drivers or VirtualBox. dkms recommends virtual package linux-image provided by each and every linux-image-<version>-{generic,server,goldfish,lowlatency}. That prevents their autoremoval as long as dkms is installed and apt is configured not to autoremove recommended packages (Apt::AutoRemove::RecommendsImportant).

[Sergey Romanov (sml-uni)]

Correction.
APT::AutoRemove::RecommendsImportant and APT::AutoRemove::SuggestsImportant are only used by aptitude. apt-get just recycles the values of APT::Install-Recommends and APT::Install-Suggests when deciding which packages should be autoremoved. apt-get's behaviour is also affected by an undocumented and unused option APT::Install-Recommends-Sections that allows autoinstalling/autoremoval of Recommends to be restricted to certain archive sections.

Nevertheless, since APT::Install-Recommends is usually set to true on Ubuntu, the point 2) from my previous comment still applies.

[Cornelis Spronk (spronkc)]

boot partition fills up with updates of kernel.

autoremove does not remove previous kernels

[jsland (greenplanet)]

Same problem here for some months now on 14.04 Studio. Autoremove and bleachbit don't deal with it. Manual cleanup around the November timeframe last year got me to the March cycle of updates before /boot was overloaded again. Problem appears to me to be centered on the fact the old ver. are not being removed completely. Using synaptic to get rid of older ver. only seems to clear a fraction of the reported image size.

[Sam (litle-sam)]

Added my +1 as it affects me to.

[remerson (ubuntu-remerson)]

This bug is tedious. /boot holds only 3 kernels for me and it seems there is a new kernel every couple of weeks. So, each time, I need to go and delete the previous 1-2 manually. Why do I have to babysit my machine? Lucky I do not want to do unattended upgrades.

Resizing /boot with LVM requires much hoop-jumping (good luck for a beginner) and only slows down how often this happens.

Please can old kernels be auto-removed properly (And/or /boot sized a bit more sensibly by default?).

There is a "latest" kernel metapackage. Can we have a "previous1" and/maybe "previous2" as well - then everything else can just go, right?

[Wylbur (wilbur-wilbur)]

Increasing the boot partition does not solve the underlying problem of old kernels. Is the size large enough now to accommodate enough kernels? If it is, we should tackle the real problem of removing old kernels.

[Snoopy (snoopy-u)]

Same problem as indicated above on 14.04 LTS. Why are "old" kernels left on a drive that is too small to begin with? Thank you in advance for fixing it. If fix would allow to expand partition (at least double it) and remove "old" kernels, this would be great!

[James Donner (jamesadonner)]

Trying this on a Dell Inspiron 15 7547 running 100% Ubuntu (No dual booting). We're unable to upgrade from Ubuntu 14.10 to 15.04 due to this error. We've tried

>sudo apt-get clean
>sudo apt-get autoremove

and nothing has worked. We don't want to get left behind on software updates, so fixing the problem is a high priority.

[Rod Smith (rodsmith)]

This bug comes up again and again on Askubuntu:

http://askubuntu.com/questions/171209/my-boot-partition-hit-100-and-now-i-cant-upgrade-cant-remove-old-kernels-to
http://askubuntu.com/questions/345588/what-is-the-safest-way-to-clean-up-boot-partition
http://askubuntu.com/questions/89710/how-do-i-free-up-more-space-in-boot
http://askubuntu.com/questions/528081/boot-partition-full
http://askubuntu.com/questions/340584/boot-is-full-cant-get-apt-get-to-upgrade
http://askubuntu.com/questions/635798/boot-partition-full-failing-to-delete-old-kernels-broken-packages

It's causing a lot of people a lot of problems.

[Hedley Finger (hedley-finger)]

I have this problem but removing old kernels was only partly successful, probably because I used rm and not apt-get method. In fact, if you have a laptop, why use LVM at all? You are scarcely going to wander around with an external hard disk attached to it for the extension to a logical volume.

The installer needs to be a lot smarter about people's level of knowledge re partitions and kernels, and give options for adjusting some of the parameters, e.g. LVM v. normal partitions, size of /boot, guided removal of old kernels, perhaps keeping three consecutive updates (including the current working kernel).

I just went through a torrid attempt to use Gparted (doesn't recognise LVM lv's) and system-config-lvm (which shows the swap and boot partitions at one end of the sda5 red cylinder, and the logical view, which shows them at the OTHER end of the blue cylinder).

I am totally lost in techhead space so am resorting to a new install to "fix" this problem. I chose Ubuntu over Fedora, SuSE and all the rest because it seemed the most novice-friendly, but this ordeal has proved me wrong. 8^)

[Remun.J (remun-j66)]

Someone give me the advise to use the LVM option on a fresh Xubuntu 15.04 install so i did. Use one internal drive, a 256GB SSD and a 2TB network drive (small NAS) for external storage. Hope there will be a fix to autoremove those old kernels from /boot partitions before upgrading a new one.

[Myamlak H.A. Bladawiec (natror-croolik-sryc)]

same here, Ubuntu 15.04 64 bit, encrypted [as opposed to the LVM].
Installed about 2015/09/15, and the Update Manager refuses installing already the very first update to the Kernel package(s), i.e., from 3.19.0.28.27
to 3.19.0.30.29.

·

·

·

[I know it's off-topic here, but is there any way round except reinstalling of the whole system? Which btw I know now I have to do manually :) ]

[georgebaily (z-george)]

Firstly, me too, this is super annoying. As a rule of thumb any 'feature' that *requires* the user to drop to CLI to put in regular manual commands *to keep the system secure and updated* ... in a desktop / consumer OS... is a bug.

It seems there are separate issues here that could be addressed separately:

1. the auto setup does not intelligently ask/warn about its default of setting up a very small /boot

2. there exists no user-friendly auto cleanup for the things clogging up /boot

3. the /boot does not have any elegant way of automatically (or via a setting) "overflowing" into the (usually abundant) storage space outside of /boot for its old kernel data etc

4. when the "full, cannot update any more" problem occurs, there is no useful feedback or suggestion or wizard from the OS for the user... just essentially a FU message.

5. apparently there is not an "easy" way to resize /boot to allow more crap to accumulate and thus postpone the problem... if there *is* an easy way then see (4) it should be provided then and there to the user; if there is *not* an easy way then this proves the point that this is not a trivial bug.

Addressing at least one of these would help the problem, and all could be worked on.

[Ian Weisser (ian-weisser)]

apt-get remove/purge/autoremove is blocked by the (always-failing) kernel install (not enough space on device)

[Ian Weisser (ian-weisser)]

Ubiquity creates space for enough kernels. The system is not managing that space later. Not ubiquity's fault.

These two issues together create the problem on a small /boot partition:
apt: apt-get remove/purge/autoremove is blocked by the (always-failing) kernel install (not enough space on device)
unattended-upgrade: 'autoremove' is disabled by default.

Adding both packages to the bug report. Removing ubiquity.

Workaround - How users can recover from the problem:
- You can use dpkg to manually remove older kernels until apt has enough space to work, or
- You can use dpkg --set selections to un-select the kernel packages for install, then run apt-get autoremove

Workaround - How users can prevent recurrence after successful recovery:
- You can enable autoremoval in /etc/apt/apt.conf.d/50unattended-upgrades , or
- You can mark your wall calendar and manually run apt-get autoremove every few months

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sebastian Nohn (sebastian-nohn)]

Change summary, because /boot is always to small on "LVM or Encrypted install". The former summary read like it was a user error, while it isn't.

[Ian Weisser (ian-weisser)]

Developer of unattended upgrades has submitted a patch:
https://github.com/mvo5/unattended-upgrades/pull/19

[Achim Behrens (k1l)]

This really needs to be adressed ASAP.

The Support Communites are flooded by angry users with blocked Packagesystems due to a full /boot blocking new kernels to be installed.

In this time, when we have 5TB HDDs and even 500GB SSDs, i really dont know why we still need that small /boot partitions that it runs out of space that fast.
And having no automatic removing old kernels is another issue.

This really spoils the user experience, and non technical users really dont know what to do and dont feel comfortable to delete files in /boot manually. So best is do increase the /boot partition made on install and have a post-install script running that removes old kernels.

[Ian Weisser (ian-weisser)]

Please do not spam bug report with "me too!" or "this bug is awful!"
comments.

The bug report is a living document used by the triagers and developers
to share data, many of them volunteers, and noise makes their job
harder.

A fix has has already been made in the upstream git, and is awaiting
merge. See https://github.com/mvo5/unattended-upgrades/pull/19

[Paul Gear (paulgear)]

There are a number of bugs marked duplicates of this bug that really aren't. The size of /boot is orthogonal to the issue of purging unused kernels. 256 MB gives the ability to install approximately 4 kernels. For some of us that's not enough, and we need to specify a larger /boot. Shortly, I intend to unmark 1465050 as a duplicate, and mark 1511911 as a duplicate of that.

[Jarno Suni (jarnos)]

Ian Weisser, how is Unattended-Upgrade::Remove-New-Unused-Dependencies better than Unattended-Upgrade::Remove-Unused-Dependencies that is already available? Why not just use a specific script to remove old kernels instead of using apt-get autoremove?

[daniel smith (opened-to)]

this is affecting me too.

Filesystem Size Used Avail Use% Mounted on
udev 981M 4.0K 981M 1% /dev
tmpfs 200M 1.1M 199M 1% /run
/dev/mapper/ubuntu--vg-root 72G 7.4G 61G 11% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 997M 404K 996M 1% /run/shm
none 100M 52K 100M 1% /run/user
/dev/sda1 236M 211M 14M 95% /boot
/home/rebecca/.Private 72G 7.4G 61G 11% /home/rebecca

i thought that it was because my drive was full. i think its only maybe 40 gig?
thanks

[Sebastian Nohn (sebastian-nohn)]

Will the fix, that has been merged in the meantime make it into 14.04 LTS or do we have to wait for some future Ubuntu release?

[Jarno Suni (jarnos)]

Fixing the issue using unattended upgrades in 14.04 LTS requires fixing Bug #1492709, too.

[Jarno Suni (jarnos)]

And it still would require manually removing the kernels installed before the fix is installed.

[Ian Weisser (ian-weisser)]

Fix committed upstream https://github.com/mvo5/unattended-upgrades/commit/d6232a8b904aa2079cc959072a59e12b580252eb

[Ian Weisser (ian-weisser)]

Fix included in upstream release: unattended-upgrades 0.89 30 JAN 2016
Uploaded to Ubuntu 16.04 on 02 FEB 2016.

   * Add `Unattended-Upgrade::Remove-New-Unused-Dependencies` that
     defaults to "yes". This ensures that older kernel
     get automatically cleaned up and /boot will not overflow.
     LP: #1267059

Marking this bug as a duplicate of LP: #1267059 so it gets closed automatically when appropriate.

[Jarno Suni (jarnos)]

LP: #1267059 is about automatic removing fails even if
Unattended-Upgrade::Remove-Unused-Dependencies "true";
It was fixed already in unattended-upgrades - 0.76ubuntu1.2. There is an error in the changelog of unattended-upgrades. This is not a duplicate of bug #1267059. This bug is about making the autoremoval of excessive the default behavior.

[Jarno Suni (jarnos)]

Can someone attach the current default /etc/apt/apt.conf.d/50unattended-upgrades of ?Ubuntu 16.04 (clean install)? Is there "Remove-New-Unused-Dependencies" advertised in comment #36? Is there line
Unattended-Upgrade::Remove-Unused-Dependencies "true";
there and is it uncommented? What is in the Unattended-Upgrade::Allowed-Origins block? Is unattended-upgrades enabled by default: What does Updates-tab show in Software & Updates (i.e. software-properties-gtk), particularly items "When there are * updates". What does "sudo dpkg-reconfigure unattended-upgrades" show?

I upgraded from 15.10 i.e. not clean install, and there is no "Remove-New-Unused-Dependencies" in /etc/apt/apt.conf.d/50unattended-upgrades.

[Jarno Suni (jarnos)]

50unattended-upgrades does not contain mention for Unattended-Upgrade::Remove-New-Unused-Dependencies, but the default value is true. Automatic installing of security updates is enabled by default in Xenial.

[Jarno Suni (jarnos)]

Fixed in version 0.90

[fksdlöfioenvdfsdji (fksdlofioenvdfsdji)]

so, what one has to do in order to get this bug fixed?

Upgrade to 16.04 ?

any additional steps required?

on the other hand is it and if how is it possible to get it fixed without upgrading to 16.04 ?
cheers

[Jarno Suni (jarnos)]

For 16.04:
There are no additional steps required that I know.

For older releases;
See https://help.ubuntu.com/community/Lubuntu/Documentation/RemoveOldKernels

[Jarno Suni (jarnos)]

Except upgrading to newer release by an installation media may leave old kernels that have to be removed manually: Bug #1586303

[Ta (talu4513)]

so i still have this bug how am I meant to fix this? Am I meant to follow the work around or do I need to upgrade my system to the latest version of ubuntu?

[Ta (talu4513)]

I followed the workaround steps and they didn't work I removed the only unused header and it still says its installed?

dpkg: warning: ignoring request to remove linux-headers-4.2.0-30-generic which isn't installed
rc linux-image-3.19.0-25-generic
rc linux-image-3.19.0-51-generic
pi linux-image-3.19.0-59-generic
rc linux-image-4.2.0-27-generic
ii linux-image-4.2.0-30-generic <--- i removed the previous good old kernel and there is still not
enough space. This is an absolute joke. Why was the boot partition given so little space?

How am i supposed to upgrade my system now. Do I need to backup everything and reinstall the latest version of ubuntu for a usb?

[Ta (talu4513)]

also I noticed there are gzip archives in the boot folder can I manually remove these?

[Ta (talu4513)]

Ok I managed to resolve this issue:

opened file explorer as root using gksudo. I navigated to the boot folder then manually deleted all the gzip archives ( because I expect they would not be used).

[Jarno Suni (jarnos)]

Ta, dpkg tells the header is not installed. If you have separate /boot partition, the existence of headers do not matter because they are not stored in /boot. If the current instructions do not work, please comment at http://askubuntu.com/a/731791/21005 to tell what exactly went wrong.

[Jarno Suni (jarnos)]

And use dpkg directly only, if apt-get (or purge-old-kernels) fails.

[Donat (donat-callens)]

Thanks for the update.

On Sat, 11 Jun 2016 at 03:00 Ta <email address hidden> wrote:

> Ok I managed to resolve this issue:
>
> opened file explorer as root using gksudo. I navigated to the boot
> folder then manually deleted all the gzip archives ( because I expect
> they would not be used).
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (237035).
> https://bugs.launchpad.net/bugs/1357093
>
> Title:
> Kernels not autoremoving, causing out of space error on LVM or
> Encrypted installation or on any installation, when /boot partition
> gets full
>
> Status in unattended-upgrades package in Ubuntu:
> Fix Released
>
> Bug description:
> Currently if one chooses to use LVM or encrypted install, a /boot
> partition is created of 236Mb
>
> Once kernel updates start being released this partition soon fills
> until people are left unable to upgrade.
>
> While you and I might know that we need to watch partition space, many
> of the people we have installing think that a windows disk is a disk
> and not a partition, education is probably the key - but in the
> meantime support venues keep needing to deal with the fact the
> partition is too small and/or old kernels are not purged as new ones
> install.
>
> For workaround and sytem repair, see
> https://help.ubuntu.com/community/Lubuntu/Documentation/RemoveOldKernels
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1357093/+subscriptions
>
--
Donat

[Jarno Suni (jarnos)]

The workaround page says: "Warning: Do NOT use the 'rm' command to delete files that were placed by the package manager, including kernel files. It merely creates a new headache for you to solve when the package manager cannot remove packages due to 'file not found'. Always use the package manager to remove files that were placed by the package manager."

[Ta (talu4513)]

Well when I attempted the workaround I got this:

sudo apt-get autoremove --purge
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.

I actually used the above command to clear up space ages ago however it no longer worked. The guide said to remove old kernels and leave the latest kernel as a backup by using dpkg. But there was nothing to remove. Thats why I looked in boot partition and checked for junk. Maybe dpkg is not removing the gzip archives after a kernel is removed?

If you think I damaged my system, do you think I should reformat my system?

[Ta (talu4513)]

Hi also for me

sudo purge-old-kernels
sudo: purge-old-kernels: command not found

purge-old-kernels does not work.

[Steve Sims (steve-sims-au)]

sudo purge-old-kernels exists for me (Ubuntu 15.04) but fails because ... /boot is full :p

[snip]
update-initramfs: Generating /boot/initrd.img-3.19.0-50-generic
gzip: stdout: No space left on device
E: mkinitramfs failure cpio 141 gzip 1
[snip]

This is such a long-standing issue, have to wonder why?

[Jarno Suni (jarnos)]

Steve Sims, that is why you have to use dpkg first like told in the "Safely removing old kernels" section of the guide.

[Jarno Suni (jarnos)]

Ta, "sudo apt-get autoremove --purge" only purges automatically installed packages, but in Trusty the linux kernel packages are likely to be marked as being manually installed.

If you mean /boot/initrd.img* files by gzip archives, they should be removed automatically when you purge the respective kernel package such as linux-image-3.19.0-59-generic by dpkg (or by apt).

If you have kernel junk in /boot even if dpkg does not list the kernel(s) as being installed, there may be a bug similar to Bug #1586303. You may write a separate bug report about it.

If you use Trusty or later, run purge-old-kernels without sudo in shell to get to know which package to install, if the command is not found.

[Ta (talu4513)]

thanks jarno thats was my problem. When i run the command without root it prompts me to install additional packages.

[Jarno Suni (jarnos)]

Anyone else affect Bug #1603620 in Xenial?

[Jarno Suni (jarnos)]

In 16.04, unattended-upgrades installed kernel linux-image-4.4.0-31-generic. Now /etc/apt/apt.conf.d/01autoremove-kernels protects releases 4.4.0-28-generic and 4.4.0-31-generic, but unattended upgrades did not purge two older kernels there are in my system. All of the kernels are automatically installed. I suppose unattended-upgrades should purge extra kernels right after it installs new one.

[Jarno Suni (jarnos)]

Oh, in contrary to previous comment, for some reason, kernel 4.4.0-28 was marked as manually installed. I changed it to be automatically installed.

[Jarno Suni (jarnos)]

Can someone tell if unattended-upgrades has purged any older kernels in Xenial? Provided that security updates are configured to download and install automatically in Software & Updates, and there is line

//Unattended-Upgrade::Remove-Unused-Dependencies "true";

or

Unattended-Upgrade::Remove-Unused-Dependencies "false";

and not line

Unattended-Upgrade::Remove-New-Unused-Dependencies "false"

in /etc/apt/apt.conf.d/50unattended-upgrades

[Jarno Suni (jarnos)]

Can someone confirm that security updates are installed automatically by default in Xenial?

[Ta (talu4513)]

sorry guys one day, I updated and during the update it said my boot folder had zero space. Then when i restarted I couldn't boot. I managed to use an older version of the kernel in order to login and backup my files thankfully.

I think I will uninstall this version of ubuntu. Hopefully this issue is fixed in a later version of ubuntu...

This time I will remember not to accept the non-defaults and I will allocated like 5 gigs to the boot sector...

[Jarno Suni (jarnos)]

Ta, which version of Ubuntu do you mean? Did you try the workaround mentioned in the bug description?

[Jarno Suni (jarnos)]

Ta, oh, it seems you have tried to apply the workaround earlier. If you removed extra kernels then and setup Unattended Upgrades to install and remove kernels automatically, /boot folder should have some space, so I don't understand why it was full again.

[Jarno Suni (jarnos)]

Ta, can you upload your 10periodic, 20auto-upgrades and 50unattended-upgrades from /etc/apt/apt.conf.d? and some log file from /var/log/unattended-upgrades that tells about kernel upgrades?

[Mathew Hodson (mhodson)]

This was fixed in Xenial. The default is now to remove newly unused dependencies after every unattended upgrade.

---
unattended-upgrades (0.89) unstable; urgency=medium

  [ Michael Vogt ]
  * Add `Unattended-Upgrade::Remove-New-Unused-Dependencies` that
    defaults to "yes". This ensures that older kernel
    get automatically cleaned up and /boot will not overflow.
    LP: #1267059
  * Remove downloaded deb packages after successful installs.
    This can be controlled via the option:
    `Unattended-Upgrade::Keep-Debs-After-Install`
    (Closes: #809428)
  * Only remove debs in the cache dir

  [ Steffen Köhler ]
  * Allow to configure sender email via `Unattended-Upgrade::Sender`

  [ Antti Riikonen ]
  * Minor typo fix in comment

  [ Cleto Martin ]
  * bugfix: non-ascii chars in dpkg log file crashes unattended upgrades
    Closes: #812857

  [ James Valleroy ]
  * Set debconf value for auto updates based on current configuration.

  [ Alexandre Detiste ]
  * update French transalation

 -- Michael Vogt <email address hidden> Sat, 30 Jan 2016 16:04:52 +0100

[Mathew Hodson (mhodson)]

I don't think backporting the new defaults to Precise and Trusty would be right for an SRU.

On Precise and Trusty, you can set Unattended-Upgrade::Remove-Unused-Dependencies "true";

This is false by default. Setting it to true will remove *all* unused dependencies after every unattended upgrade.

[Jarno Suni (jarnos)]

Unattended upgrades does not remove old kernels that were installed by other means than unattended upgrades even if "Unattended-Upgrade::Remove-Unused-Dependencies "true";" is set in Trusty (and maybe in Precise) Bug #1492709

[Jarno Suni (jarnos)]

I think this `Unattended-Upgrade::Remove-New-Unused-Dependencies` is not well tested yet. I have not noticed if it has removed any kernel. Also, I think it will not remove a kernel, if you happen to install a new kernel using Software Updater before Unattended Upgrades operates.

[Ian Weisser (ian-weisser)]

There seems a lot of confusion in this bug report about which pieces of the system work together to cause this problem, and which pieces should be responsible for fixing it.

Marking kernels as eligible for autoremoval is done by apt. The apt package provides /etc/kernel/postinst.d/apt-auto-removal, which sets the rules for marking kernel packages eligible for autoremoval. If you discover a system that keeps more than three kernels (and isn't already broken), run 'apt autoremove' manually. If the system still has more than three kernels, then see if you can figure out why and then file a bug against Apt.

But apt only marks. Apt doesn't actually pull the trigger and run an autoremove.

Compounding the problem is another issue with apt: Apt in 16.04 and earlier queues autoremoval instead of running it first (or last...or scheduled). This is what makes broken systems harder to fix. Out-of-space hits before autoremove.

Unattended-Upgrades prior to 16.04, if the option is enabled, merely runs 'apt autoremove' via aptdaemon. U-U does not check nor care what packages need to be removed. U-U does not check nor care if the system is LVM/Encrypted or has a small separate /boot partition. None of that is part of U-U's job.

U-U developers in 16.04 addressed this problem (even though it's arguably not U-U's problem) by checking if packages are already marked eligible-for-autoremove at the beginning of the run. If so, it autoremoves them before doing anything else. Again, U-U doesn't care about which packages happen to be marked, or wheter the system has a small /boot partition.

In 16.04, the problem is undeniably fixed. The additional functionality of U-U has prevented stale kernels from accumulating on all my test systems. However, developers have understandably shown little interest in backporting that particular fix.

And there are other options to integrate the fix better into apt itself instead of U-U. For example, aptdaemon does seem to have queue management tools - it should be possible to move an 'autoremove' to the head of queue (incidentally making fixes much easier). Or we can add another kernel hook file to /etc/kernel/postinst.d/ to trigger an autoremove upon kernel installs (instead of U-U's daily check). Or we can add a startup/cron.daily service that simply checks for existence of a small /boot partition and runs autoremove before (or after) the daily apt update. Or we can have aptdaemon catch out-of-space errors (instead of simply passing them), run autoremove, and then try the install again.

Almost none of these possible solutions would be implemented in Unattended Upgrades.

[Brian Candler (b-candler)]

I have two Precise (12.04) servers with

Unattended-Upgrade::Remove-Unused-Dependencies "true";

in 50unattended-upgrades. One of them cleans up its kernels and only keeps the last two; one of them accumulates kernels over time, and I occasionally get alerts about /boot filling up.

On the latter machine, "apt-get autoremove --purge" doesn't remove them. I end up removing the packages individually; the script "purge-old-kernels" referred to above doesn't seem to exist.

The main difference I can see is that the latter machine has linux-image-generic-lts-trusty, whereas the first machine has linux-image-server.

Both machines have in /etc/apt/apt.conf.d/05aptitude:

aptitude::Keep-Unused-Pattern "^linux-image.*$ | ^linux-restricted-modules.*$ | ^linux-ubuntu-modules.*$";

So I don't think that's it. I suppose there won't be too many kernel updates for Precise before it goes end of life now though...

[Jarno Suni (jarnos)]

Brian Candler, check if you have same repositories enabled in both servers. You should get purge-old-kernels, if you add byobu PPA repository.

[Jarno Suni (jarnos)]

Ian Weisser, I think in 16.04, in contrary to what you claim, packages are not autoremoved before doing anything else. https://github.com/mvo5/unattended-upgrades tells:
"Unattended-Upgrade::Remove-New-Unused-Dependencies - boolean (default:True)

Remove any new unused dependencies after the upgrade finished."

Anyway, in my 16.04 system, where I have the setting enabled (as default) and unattended-upgrades enabled (via Software & Updates), I currently have 6 kernels installed, so I think it is not doing its job well.

[Ian Weisser (ian-weisser)]

Jarno Suni,

Okay, I stand corrected on that statement.

Please explain how you determined that Unattended Upgrades is the culprit, instead kernel header packages or user changes to apt-marking or other possible causes for the symptom you describe.

With the information so far, I cannot duplicate your issue on my test system. How can I do so?

[Jarno Suni (jarnos)]

I guess the reason is I have installed some kernels using Software Updater even if Unattended Upgrades is set to install security updates. I guess I am not the only one using update-notifier and update-manager to install updates.

[Ian Weisser (ian-weisser)]

Jarno Suni,

That possible cause seems irrelevant to Unattended-Upgrades. Any
package installed by any front-end to apt will be upgraded. U-U doesn't
know what packages are installed, nor care. U-U simply tells aptdaemon
to upgrade packages from the repositories authorized by the user. U-U
does not duplicate nor replace Software Updater, and both can safely be
used.

Try running 'apt autoremove' (16.04) or 'apt-get autoremove'
(12.04/14.04).

If those kernels are not removed, then your problem is the apt-marking,
not U-U. There are several ways to prevent automatically marking kernel
packages as 'auto' - the most common is by installing kernel header
packages.

If those kernels are removed, then check the U-U logs in /var/log to
ensure that U-U is really operating.

A bug report is an awful place to try to troubleshoot in this way. If
you open a thread on UbuntuForums, we may be able help you better.

[Jarno Suni (jarnos)]

Ian Weisser, the issue is covered by the title of this bug report, anyway. If 'Unattended-Upgrade::Remove-Unused-Dependencies "true";' is used, U-U removes extra kernels, even if new kernels are installed by Software Updater in 16.04. In 14.04 (and in 12.04 I suppose) it does not work due to bug Bug #1439769. So running 'apt-get autoremove' in 14.04/12.04 may not have the desired effect. Anyway, the point of this bug report was that old kernels should be autoremoving so that administator does not have to be concerned of /boot getting full, so I think another package should be added as being affected, too.

How does installing kernel header packages (which do normally get installed anyway) prevent marking kernel packages as 'auto'? I do not see that happening in 16.04 at least. linux-header packages do not depend on linux-image packages besides linux-header packages may itself be marked 'auto'.

[Ian Weisser (ian-weisser)]

Jarno Suni,

This bug report ran it's course, and 'Fix Released' is another way of
saying 'Closed.' You can discuss it, but no developer is likely to read
it...to them, it's a closed issue. It's fixed, and won't be unfixed.
The patch is released, and won't be reverted. The normal workflow is to
file a new bug, not to reopen or discuss a closed bug.

Bug 1357093 is 'Fix Released' because I duplicated the original
reporter's problems, triaged the bug, and forwarded enough information
to a developer who agreed with my triage and decided to fix it. Upon
testing the fix (repeatedly), my duplication of the original reporter's
problem is really fixed.

If you have a triage-able problem with the unattended-upgrades package,
please open a new bug report and provide enough information so I can
triage it. I cannot generally triage a problem that I cannot duplicate.

If all you know is 'it should work but does not work', then please jump
into a support forum (like Ubuntuforums) and let the experts there help
you figure out how to duplicate or triage it. Similar symptoms may come
from very different causes. Having an unresolved problem is really,
really frustrating. Been there.

I'm quite happy to discuss kernel header dependencies and go into much
greater detail...but doing so in this bug report would be rather
counterproductive. The issue is closed, and most of the questions seem
unrelated to the original bug.

[Jarno Suni (jarnos)]

This bug is 'Fix Released' because I marked it as such. Maybe too soon as it is irreversible.

Ok, if someone writes new bug report about kernels not autoremoving, please link it here.

Ian Weisser, you may e.g contact me privately, if you want to tell me about the kernel header packages thing.

[Jarno Suni (jarnos)]

Bug #1624644 claims U-U does not work well with Software Updater a.k.a. update-manager. Can you confirm?

[fksdlöfioenvdfsdji (fksdlofioenvdfsdji)]

evendough people talk about not adding more comments to this thread.

i had the problem, then they/you/here they said solution is upgrade to 16.04

- i solved the problem manually und than upgraded to 16.04

- now some months later i wanted to update && upgrade my server and boot is full again. - and i had to solve it manually again.

so for my experience > fix didn't work.

what i did is:
- nothing
- just have the unatended auto updater install security updates. (configured as in 14.04) super default basic.

i now loocket in to the "50unattended-upgrades" file and

// Do automatic removal of new unused dependencies after the upgrade
// (equivalent to apt-get autoremove)
// Unattended-Upgrade::Remove-Unused-Dependencies "false";

is commented..
Would i need to change that? or is that the unrelated one?

[Jarno Suni (jarnos)]

fksdlofioenvdfsdji,
How did you "want to update && upgrade your server"?
How many kernels can your /boot partition hold? (What is the size of your /boot partition?)

Contents of "50unattended-upgrades" is misleading, as there is no mention of the new configuration variable Unattended-Upgrade::Remove-New-Unused-Dependencies which defaults to "true" in the file (in Ubuntu 16.04).

Replacing line
// Unattended-Upgrade::Remove-Unused-Dependencies "false";
by
Unattended-Upgrade::Remove-Unused-Dependencies "true";
may help you, if you don't mind that it removes all packages that 'sudo apt-get autoremove' would.

[Not Martin Wimpress (not-martinwimpress)]

Does anyone have this problem in Linux Mint 18? "Unattended-Upgrade::Remove-New-Unused-Dependencies" also does not exist for me but the package was last updated a month ago, which leads me to believe the configuration file "50unattended-upgrades" wasn't "replaced" with the new version. Also, the package has other issues including not fetching from the correct repos as it's a stock Ubuntu config.

[C Filorux (breakfast)]

The fix is failing in xenial: I have to repeatedly clean up after the installer, when it complains that it cannot do its work due to disk space on the tiny /boot partition. "My disk is full again!"

The bug remains: The automatic update is not considered an unattended update, since the user is there attending it via an interactive application (tell me I'm wrong).

[Ian Weisser (ian-weisser)]

Another (complimentary) solution for accumulating kernels in an LVM environment: http://blog.surgut.co.uk/2016/11/boot-less-lvm-rootfs-in-zesty.html

The fix to this bug prevents kernels from accumulating.
That complimentary solution eliminates the tiny /boot partition, eliminating the space problem on LVM desktops and laptops...though not on other space-constrained devices.

My updated u-u on Ubuntu 16.04 and 16.10 is working just great, and correctly preventing kernels from accumulating. However, we have seen similar-looking apt problems in the support forums: A few systems to continue to run out of /boot space. Upon investigation, these are usually due to unconfigured/half-configured packages or user confusion with the settings.

If your unattended-upgrade is not working properly, and your system continues to run out of /boot space, please seek help in a support forum before posting permanently to this CLOSED bug report.

[Mossroy (mossroy)]

Ian, could you please give the URLs where to find the "similar-looking apt problems in the support forums" you're mentioning?
I think many people here (including me) would be interested in what are the correct settings you mention.

[Ian Weisser (ian-weisser)]

Happy to discuss and help...in the support forums.
This CLOSED bug report is not the appropriate location to help people troubleshoot.

[Mitch Singler (msingler)]

Seems to be a design flaw rather than a bug.

      From: Ian Weisser <email address hidden>
 To: <email address hidden>
 Sent: Tuesday, November 22, 2016 9:26 PM
 Subject: [Bug 1357093] Re: Kernels not autoremoving, causing out of space error on LVM or Encrypted installation or on any installation, when /boot partition gets full

Happy to discuss and help...in the support forums.
This CLOSED bug report is not the appropriate location to help people troubleshoot.

--
You received this bug notification because you are subscribed to a
duplicate bug report (1183692).
https://bugs.launchpad.net/bugs/1357093

Title:
  Kernels not autoremoving, causing out of space error on LVM or
  Encrypted installation or on any installation, when /boot partition
  gets full

Status in unattended-upgrades package in Ubuntu:
  Fix Released

Bug description:
  Currently if one chooses to use LVM or encrypted install, a /boot
  partition is created of 236Mb

  Once kernel updates start being released this partition soon fills
  until people are left unable to upgrade.

  While you and I might know that we need to watch partition space, many
  of the people we have installing think that a windows disk is a disk
  and not a partition, education is probably the key - but in the
  meantime support venues keep needing to deal with the fact the
  partition is too small and/or old kernels are not purged as new ones
  install.

  For workaround and sytem repair, see
  https://help.ubuntu.com/community/Lubuntu/Documentation/RemoveOldKernels

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1357093/+subscriptions

[Mossroy (mossroy)]

For those still having this issue on Ubuntu 16.04 : please check that you did not disable the automatic installation of updates.
It might be the reason why the old kernels were not removed for me (not 100% yet). See discussion in https://ubuntuforums.org/showthread.php?t=2344232
With a GUI, it's in System settings->Software and updates->Updates tab : "When there are security updates" should be set to "Download and install automatically" (its default value).
(please comment on the forum thread)

[Henry (hvanmegen)]

I can't even run `apt-get autoremove` because it starts filling up my /boot with old kernels until it runs out of space.. who the hell designed this madness?

[Henry (hvanmegen)]

Ok, I had to manually remove the files as they were added.. after that, it worked as intended.

Funny thing is, all solutions provide here say that you need to enable the removal of unused dependencies by setting this:

`Unattended-Upgrade::Remove-Unused-Dependencies "true"`

Problem is.. I already had this set in my `/etc/apt/apt.conf.d/50unattended-upgrades`..

This is broken, design or not.. it IS a bug.. how can we fix this permanently? I have servers running unattended upgrades that have overflowing /boot partitions which is annoying AF.

[Jarno Suni (jarnos)]

Henry, do you use Ubuntu Trusty, and install kernels manually? See comment #79.

[Bryce Nesbitt (bryce2)]

My elderly parents are on Ubuntu 16.04, with updates set to 'Download and install automatically'.
I must manually prune the Kernel images every so often, or the system's underwear gets all in a twist, and "apt-get autoremove' won't even work.

Let's be clear: THIS IS NOT FIXED as of 16.04 LTS.

[Bryce Nesbitt (bryce2)]

Not fixed
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1675079

[spike speigel (frail-knight)]

This might be aggravated by this DKMS bug which was found in v2.2.0.3, which incidentally is used in Ubuntu 16.04 LTS, and maybe other versions. It doesn't remove old initrd files in /boot *which lead to full /boot issues and subsequent update problems*

I manually removed a bunch of old initrd images the other day.

http://askubuntu.com/questions/616512/purging-old-kernels-fails-to-remove-old-initrd-files

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717584

[spike speigel (frail-knight)]

Found a DKMS bug report here:

https://bugs.launchpad.net/ubuntu/+source/dkms/+bug/1515513

[James Duvall (jamesduvall)]

running 16.10, I am also running in to the problem that autoremove fails to remove old initrd files. This was greatly exacerbated by bug #798414 (https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/798414). With an encrypted file system and a small boot partition, I only have space for 3 kernels. With the pace of new kernel builds, it doesn't take long to run out of space. Now that I understand the problem, it is not such a big deal, but it was very annoying to have unhelpful initramfs errors nearly every time I boot up.

[Jackalux (heartwoodjack)]

I don't have hardware that I imagine will run 16.xx so need this addressed on older LTS versions.

My output for...
dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d'
 ... was 120 lines long! I had to teach myself scripting to delete them all manually. I only discovered the error after a different Linux machine I created (with a separate /boot) ran into the bug.

Deleting this junk freed 10GB of total disk space.

This is too much to ask of novice users and will stop Linux being more widely adopted. I'm not a complete Noob but this was annoying to deal with... it will be impossible for others.

[Andreas Lindhé (lindhe)]

I hate to be that guy that complains without contributing in a constructive manner, but I just need to say that it's completely unreasonable that my mom and friends I recommend Ubuntu to gets this problem. They should never have to be concerned with what a kernel image is or why the boot partition is full.

[Jarno Suni (jarnos)]

Jackalux, oh I see you have found the popular one-liner script for kernel removing. It is too complicated for what it does and too simple for what it should possibly do.

I have written a script to ease kernel removal:
https://www.bountysource.com/issues/38300038-feature-request-the-command-should-work-like-this
You (and other complainers) could post a small bounty to fund the work. Alternatively I think we could form an association to fund the work and other stuff that novice Ubuntu users may need provided it could attract enough paying members.

Andreas Lindhé, you have a point. I don't know who decides it, but I suppose the fix to the bug could be backported or SRUed to even older LTS releases. But it seems that the fix does not work even in 16.04 (Bug #1675079); that applies only to the new Unattended-Upgrade::Remove-New-Unused-Dependencies setting in unattended-upgrades, though, so if setting Unattended-Upgrade::Remove-Unused-Dependencies "true" suits to you, there is a fix available. Alternatively, e.g. my script could be run from a cron job or (in 16.04 or later) from a systemv timer to handle automatic kernel removal.

[Jarno Suni (jarnos)]

Correction: I meant systemd, not systemv in the previous comment.

[cmp (cmp001)]

I had previously recommended (and consequently installed for them) Ubuntu 16.04 LVM to my parents however their"/boot" partition was constantly filling up to 100% and they kept complaining to me about it (as they are computer illiterate) so I had to eventually reinstall windows 10 for them and scrap Ubuntu all together. A pretty sad state of affairs. This is definitely a bug and it is making people turn away from Linux in favor of Windows. I wish someone would provide a simple fix for this.

[Andreas Lindhé (lindhe)]

Is this bug confirmed resolved for any version of Ubuntu? What release do I need to get my friends migrated to, in order for this to not be a problem?

[Jarno Suni (jarnos)]

No. Automatic removing is supposed to work in 16.04 and later by default. But even in 16.04 it works only, if you let unattended-upgrades install all kernel updates. I have not tested later releases, but I guess it is the same thing.

I think removing should work in 16.04 and later, if you use the configuration recommended in https://help.ubuntu.com/community/RemoveOldKernels. (i.e. setting Unattended-Upgrade::Remove-Unused-Dependencies "true";) This configuration works by 14.04, too, if you let unattended-upgrades install all kernel updates.

[Stuart Bishop (stub)]

It also requires a complete reinstall if your boot partition is too small (ie. you accepted the defaults with an LVM or encrypted disk install pre 16.04 release). Machines upgraded from 14.04 will continue to fail and be directed here.

[Dave Eckhardt (de0u)]

I tried the advice given on https://help.ubuntu.com/community/RemoveOldKernels, but it mostly doesn't work. For example, "apt-get autoremove --purge" removed zero kernels. To install "purge-old-kernels" would require installing like 60 dependencies of "bikeshed", which seems unreasonable.

I think part of the problem is that the tools behave differently depending on who installed a kernel, but /boot gets full based on the number of kernels, not who installed them. My machine (a netbook) is basically never left on long enough to run an unattended upgrade, so I run upgrades manually. That means basically every kernel was manually installed, so something that removes only automatically installed kernels is 0% useful. In the real world the problem is disk space, not assigning blame for who installed a given kernel! Any process that installs new kernels without deleting old kernels will eventually fill up a finite-sized /boot. Any solution that doesn't address the sad fact of the finiteness of /boot will leave some people, probably the most inexpert, stuck. I personally can manually grub around and remove kernels, so this is only annoying, not show-stopping, but I think we have lots of evidence that it is show-stopping for some people, which suggests that there are many more people out there who don't know how to complain who are also injured.

Bottom line: /boot is finite, so anybody who installs a kernel without deleting a kernel is doing something that will sink somebody's ship. Given that lots of machines out there are clogged by the old maintenance processes, any solution that doesn't explicitly address how to un-clog a machine clogged by the previous solutions isn't really a solution.

[Jarno Suni (jarnos)]

stub,

I suppose the upgrading process did not change the settings concerning unattended-upgrade. You should do that manually after upgrading from 14.04 and initially remove extra kernels manually before upgrading.

de0u,

I tell about the restrictions concerning 'apt-get autoremove' in the wiki page. Likewise I tell about some alternatives not limited to 'purge-old-kernels'. I think it is harsh to say most things do not work, if you do not specify.
You could run Unattended-upgrade from command line to install new kernel updates when you want:

sudo apt-get update && sudo unattended-upgrade

As for un-clogging, did you read the 'Safely Removing Old Kernels' chapter? I see many people take time to complain, but I see none who is willing to spend even 1$ to get an easier solution for poor Ubuntu users.

[David Potter (akxw-hzxb-j0p9)]

I'm running Ubuntu 17.04 and whilst I have had the problem for so long I can't be sure, I think I have had this problem since 15.04. I keep my software completely up to date and all updates have been applied to date. I can honestly say that this issue is by no means fixed. The reference to unattended-upgrades is a red herring. I boils down to a faulty kenal cleanup routine and a poor default for the boot partition at install time.

I have over the many months of experiencing the problem, finally come up with the following work around set of commands that free up the space required to keep me going with a few more updates...

sudo apt-get autoremove;sudo apt-get autoclean;sudo apt-get clean

dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d' | xargs sudo apt-get -y purge

sudo apt-get update && sudo apt-get upgrade

Although this sorts the problem out when I get the "not enough space" issue it doesn't mean that the underlying problem shouldn't be sorted out.

It takes time and effort that shouldn't be required.
It has taken many months to come up with a decent work-around.
I can't use this type of install for family and friends as they are not savvy enough to use the work-around and shouldn't need to be using sudo in the first place.

I see this issue is marked as wishlist. This is a travesty. It is a bug pure and simple.

[Sebastian Nohn (sebastian-nohn)]

alias remove-old-kernels='dpkg -l linux-{image,headers}-* | awk '\''/^ii/{print }'\'' | egrep '\''[0-9]+\.[0-9]+\.[0-9]+'\'' | grep -v 4.4.0-79 | xargs sudo apt-get -y purge'

[Dave Eckhardt (de0u)]

jarnos,

The reason why I responded first/primarily about "apt-get autoremove" and "purge-old-kernels" is that those are listed at the top of the page and are the things that a regular-ish user could feel confident doing.

I do not feel that the idea of running unattended-upgrade from the command line is likely to be helpful to many users. Among other things, that presumes that the user always gets in there before being prompted by the automatic update GUI program (which, by the way, doesn't say "I am going to install a kernel"; it says "I have 192 MB of security updates that I would like you to approve", unless you dive into details).

As for un-clogging, I did in fact read the "Safely Removing Old Kernels" text. What I wrote was "I personally can manually grub around and remove kernels, so this is only annoying, not show-stopping, but I think we have lots of evidence that it is show-stopping for some people". I continue to believe that.

I did not see a response from you to what I genuinely believe is the single core issue: anything (unattended-upgrades, GUI install widget, apt, flying spaghetti monster) that installs a kernel without deleting a kernel will eventually overflow /boot.

Then there is a second issue, which is that because a variety of things have installed too many kernels on the machines of people who cannot be expected to manually delete kernels, there should be some automatically-run solution (perhaps prompting the user for confirmation for safety purposes) that deletes long-stale kernels that no current automatic thing considers itself responsible for.

What I see is:
(1) new users keep showing up because their machines have broken,
(2) people keep proposing new solutions that are different than the existing solutions.

To me this says that there is still a real problem (for some people, not for me, but I don't think the aspiration of the Ubuntu project is for machines running Ubuntu to work only for expert users).

Thanks for taking the time to respond! I am not responding because I am personally troubled by this issue, but because I believe that the problem has not been solved for regular end users.

[Mitch Singler (msingler)]

Well said Dave! It's hard for me to understand why smart people don't understand this issue. This required maintenance should not even be required for expert users let alone the average user. In over 30 years in IT and multiple platforms I have never heard of this design flaw/systems implementation being done on purpose. Perhaps someone can explain the design purpose that creates this issue as I have not seen it any where in these forums and bug reports.

[Andreas Lindhé (lindhe)]

I'm completely with Dave on this one.

Regarding jarnos comment #110:

> I see many people take time to complain, but I see none who is willing to spend even 1$ to get an easier solution for poor Ubuntu users.

Is funding the problem here? Because if it is, please tell us. I'm sure we can at least gather some funds, since this is such a prevalent issue that clearly stirs a lot of people into motion.

[fksdlöfioenvdfsdji (fksdlofioenvdfsdji)]

Just one quick comment that issue for me never got fixed, and i switched to debian. and will finally hopefully find a way to stop getting notified about comments on this thread.

[Jarno Suni (jarnos)]

The reason why 'apt-get autoremove' is stated there is that it is a simple working way in 16.04 and later for most users, if your system is not broken already. purge-old-kernels is included in some Ubuntu package; consequently it is supported by Launchpad's bug reporting system. But as said in the document, it is currently deprecated and its bugs will likely not be fixed. It might still be useful for some users in some cases.

If one wants to use unattended-upgrades manually, he/she could disable it from being called automatically. One should avoid installing security updates manually by Software Updater in 14.04, if one expects to use apt/unattended-upgrades for removing kernels automatically.

I have developed some software that makes the un-clogging part easier; link to related bountysource.com page is included in the document. Just feel free to post a bounty and/or give some feedback about the interface in the respective Launchpad page. I guess the software could be launched automatically in case of kernel clog when it happens in interactive session (or even in non-interactive session, if the software is made to support completely non-interactive un-clogging).

If you install a kernel manually in command line by say
sudo apt-get install linux-image-4.4.0-87-generic
which kernel(s) that command should remove in turn in your opinion? Or could it be that the user who runs the command should take care of it? User possibly wants to keep some spare kernels, in case the one that is just installed is bad. Or a developer may want to keep some testing kernels etc.

[David Potter (akxw-hzxb-j0p9)]

Hi Jarno

I would suggest that most desktop users get notified by the updater that there are updates and run the updater accordingly to apply them. As such, I think the unclogging activity needs to run automatically as the users that are using the default boot partition size almost certainly have no interest in having to run the software themselves or even making a decision about how many old kernels to keep. In fact I suggest that there are a goodly number of users that wouldn't know what to do with an old kernel anyway.

My suggestion would be that there should be a default number of kernels to keep (2 or maybe 3 would be my vote) but have a configuration file that would allow the number to be altered for those users that wanted to change the default. In fact you could even use a 0 value to indicate that the user does not want the un-clogging to happen automatically.

It seems like all the commands and scripts have been sorted out over time and the work is really to apply them in an automated fashion. I do recognise that, there may be a more elegant solution that might be preferable though. In which case that is going to be more work.

If we are talking about putting this in place to run automatically and definitely removing the "out of space" issue with no effort on behalf of the user, then I would be happy to contribute towards the bounty. Can you confirm that this would be the intention?

I don't think that the default size of the boot partition should be ignored though. Surely it's not going to break anyone's system to up the size to 500Mb. Whilst I'm not a developer, I can't believe it would take much effort to increase the default size.

Thanks for your persistence and help moving this bug along Jarno.

[Jarno Suni (jarnos)]

Hi David

One option is to install 16.04 or later and configure unattended-upgrades as recommended in the document. I am afraid all the related bugs will not be fixed in 14.04.

Fixing Bug #1460396 and Bug #291342 would also help users of Software Updater even if unattended-upgrades is not used.

The changes you suggest have nothing to do with unattended-upgrade (against which this bug is reported).

IIRC the default separate /boot size is 500MB in some newer release of Ubuntu.

[Jarno Suni (jarnos)]

As for default settings for unattended-upgrade, it is interesting what Canonical Foundations Team is going to do with Bug #1624644.

[Jarno Suni (jarnos)]

I wonder why fix to Bug #1439769 has not been released as SRU for 14.04 yet. The fix would make unattended-upgrade work better in removing extra kernels in 14.04.

[Jarno Suni (jarnos)]

... and 'sudo apt-get autoremove' would work, too.

[Ben Norris (norro)]

This bug has just hosed a system I installed for my parents. An unattended-upgrader that breaks your system if you don't attend to it IS broken.

Having known about this problem for many years, the least you could have done is added a warning to the installer that choosing lvm or encryption will make a system that breaks itself.

The idea that novice users can just run script workarounds is entirely unrealistic.
a) They don't know about the problem if they haven't been told (burying it deep in the manuals or text that whizzes by is not good enough)
b) from a security PoV they shouldn't be run anything they cut & paste in

The general attitude to a major showstopper that is causing a lot of pain for novice users shown in this bug is frankly appalling and I'm pretty much inclined to ditch Ubuntu because of it.

[iGadget (igadget)]

Ben, I feel your pain and couldn't agree more. I'm under the impression that this attitude is getting more common as Canonical focuses on server and IoT. Where are the days where the 100 Papercuts (https://wiki.ubuntu.com/One%20Hundred%20Papercuts) got a really high priority? IMHO, this bug sure seems to qualify...

[mikewhatever (mikewhatever)]

Recently, I had to remove 42 old kernels that accumulated over about 18 months on an old PC. It only took about 80 minutes. Clearly, old kernels should be auto-purged by default, with an option to keep them available for those that need it. As is, users have to manually run apt-get autoremove, resort to script, or configure unattended upgrades, none of which should ever be expected from regular users.

[mikewhatever (mikewhatever)]

Here is a good one: https://ubuntuforums.org/showthread.php?t=2374043&p=13697015#post13697015. The OP had, in total, 74 kernels, 65 headers and 93 extras = 232 packages that almost filled /, and gobbled up inodes. Don't know if it is a record, but it's close.

[Jussi Lind (jussi-lind)]

I'm not sure how average users should handle this. I'm a professional software engineer and facing some serious dpkg issues when trying to make my 16.04 LTS work again due to filled up /boot.

[Balint Reczey (rbalint)]

This is now fixed in bionic and is scheduled for backporting to stable releases.

[luca moscato (luca-moscato)]

Sorry guys to bother you, I'm the reporter of https://bugs.launchpad.net/ubuntu/+bug/1744045

Having this bug fixed exactly what does it means on Ubuntu 16.04 and on the upcoming 18.04?
With a clean installation (on an encrypted disk) the boot partition will be removed automatically? Or do I have to upgrade manually /etc/apt/apt.conf.d/50unattended-upgrades
uncommenting this parameter?
//Unattended-Upgrade::Remove-Unused-Dependencies "true";

I'm asking since some of my colleagues doesn't use command line (so they don't use autoremove) and they rely on Software update Application.

Thanks for understanding

[Balint Reczey (rbalint)]

On Mon, Apr 23, 2018 at 5:48 PM, luca moscato <email address hidden> wrote:
> Sorry guys to bother you, I'm the reporter of
> https://bugs.launchpad.net/ubuntu/+bug/1744045

Thank you for reporting bugs!

>
> Having this bug fixed exactly what does it means on Ubuntu 16.04 and on the upcoming 18.04?

Unattended-upgrades and update-manager will remove old unused kernels
with the default configuration.
This is already happening in 18.04 and will be back-ported to 16.04.

> With a clean installation (on an encrypted disk) the boot partition will be removed automatically? Or do I have to upgrade manually /etc/apt/apt.conf.d/50unattended-upgrades
> uncommenting this parameter?
> //Unattended-Upgrade::Remove-Unused-Dependencies "true";

No, the /boot partition will be kept, but unused old kernel files will
be cleaned up. Without changing the configuration files from the
shipped default values.

> I'm asking since some of my colleagues doesn't use command line (so they
> don't use autoremove) and they rely on Software update Application.

I hope they will be pleased to see that obsolete kernels are removed
without starting terminals. :-)

[luca moscato (luca-moscato)]

Many many many thanks :)

[Kurt Fitzner (kudalufi)]

This is still broken, at least on my 18.04. apt-get autoremove does not touch old kernels. What is the process for re-reporting this?

[Balint Reczey (rbalint)]

@kudalufi If apt-get does not remove the kernels they must have been set o manually installed which prevents u-u and update-manager from removing them, too. Please remove the ones set manually installed manually.