Kernels not autoremoving, causing out of space error on LVM or Encrypted installation or on any installation, when /boot partition gets full

Bug #1357093 reported by Elfy on 2014-08-14
654
This bug affects 205 people
Affects Status Importance Assigned to Milestone
unattended-upgrades
New
Undecided
Unassigned
unattended-upgrades (Ubuntu)
Wishlist
Unassigned

Bug Description

Currently if one chooses to use LVM or encrypted install, a /boot partition is created of 236Mb

Once kernel updates start being released this partition soon fills until people are left unable to upgrade.

While you and I might know that we need to watch partition space, many of the people we have installing think that a windows disk is a disk and not a partition, education is probably the key - but in the meantime support venues keep needing to deal with the fact the partition is too small and/or old kernels are not purged as new ones install.

For workaround and sytem repair, see
https://help.ubuntu.com/community/Lubuntu/Documentation/RemoveOldKernels

Elfy (elfy) wrote :
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubiquity (Ubuntu):
status: New → Confirmed
Elfy (elfy) wrote :

Installed with LVM in a vm, /boot size 236M

/dev/sda1 236M 38M 186M 17% /boot

http://pastebin.ubuntu.com/8062443/

Installing with encryption and LVM results in the same /boot size unsurprisingly

description: updated
description: updated
summary: - Encrypted install creates too small /boot partition
+ LVM or Encrypted install creates too small /boot partition

Just dealt with the problem of an affected user here : http://ubuntuforums.org/showthread.php?t=2240240

But I guess adding myself to the "Affects Me" list would be cheating? Since it is not me who is affected.

TJ (tj) wrote :

The issue here isn't so much the absolute space initially allocated to the /boot/ LV, it is that the installer allocates 100% of the Volume Groups space.

The great advantage of LVM is that space can by dynamically allocated and removed from Logical Volumes as needed.

If the installer were to leave 5% of the Volume Group unallocated it would be possible for the system to detect this out-of-space condition and extend the /boot/ LV:

lvextend -l 50%FREE VG/boot
resize2fs /dev/mapper/VG-boot # assuming ext file-system

and at the same time display a prominent warning and options for automatically removing the superseded kernel versions.

Especially a problem when setting up systems for clients, who you only visit once or twice a year. The kernel updates fill the boot partition fast and so users are warned at the start that no space is left.

Donald Tripp (dtlongrifles) wrote :

I have this problem with Xubuntu 14.04

Changed in ubiquity (Ubuntu):
assignee: nobody → Donald Tripp (dtlongrifles)
Coffeecat (coffeecat) on 2014-12-13
Changed in ubiquity (Ubuntu):
assignee: Donald Tripp (dtlongrifles) → nobody
Sergey Romanov (sml-uni) wrote :

Link to yet another forum thread: http://ubuntuforums.org/showthread.php?t=2256479
The problem is twofold.

1) Users not paying attention to apt-get saying:

The following packages were automatically installed and are no longer required:
...
Use 'apt-get autoremove' to remove them.

like in the thread linked above.

2) Systems where linux-image-<version>-{generic,server,goldfish,lowlatency} don't get suggested for autoremoval after upgrade to newer versions of linux-{generic,server,goldfish,lowlatency}/linux-image-{generic,server,goldfish,lowlatency}. Mostly because some dkms-based kernel-modules are installed like NVIDIA/AMD propietary video drivers or VirtualBox. dkms recommends virtual package linux-image provided by each and every linux-image-<version>-{generic,server,goldfish,lowlatency}. That prevents their autoremoval as long as dkms is installed and apt is configured not to autoremove recommended packages (Apt::AutoRemove::RecommendsImportant).

Sergey Romanov (sml-uni) wrote :

Correction.
APT::AutoRemove::RecommendsImportant and APT::AutoRemove::SuggestsImportant are only used by aptitude. apt-get just recycles the values of APT::Install-Recommends and APT::Install-Suggests when deciding which packages should be autoremoved. apt-get's behaviour is also affected by an undocumented and unused option APT::Install-Recommends-Sections that allows autoinstalling/autoremoval of Recommends to be restricted to certain archive sections.

Nevertheless, since APT::Install-Recommends is usually set to true on Ubuntu, the point 2) from my previous comment still applies.

Cornelis Spronk (spronkc) wrote :

boot partition fills up with updates of kernel.

autoremove does not remove previous kernels

jsland (greenplanet) wrote :

Same problem here for some months now on 14.04 Studio. Autoremove and bleachbit don't deal with it. Manual cleanup around the November timeframe last year got me to the March cycle of updates before /boot was overloaded again. Problem appears to me to be centered on the fact the old ver. are not being removed completely. Using synaptic to get rid of older ver. only seems to clear a fraction of the reported image size.

Sam (litle-sam) wrote :

Added my +1 as it affects me to.

remerson (ubuntu-remerson) wrote :

This bug is tedious. /boot holds only 3 kernels for me and it seems there is a new kernel every couple of weeks. So, each time, I need to go and delete the previous 1-2 manually. Why do I have to babysit my machine? Lucky I do not want to do unattended upgrades.

Resizing /boot with LVM requires much hoop-jumping (good luck for a beginner) and only slows down how often this happens.

Please can old kernels be auto-removed properly (And/or /boot sized a bit more sensibly by default?).

There is a "latest" kernel metapackage. Can we have a "previous1" and/maybe "previous2" as well - then everything else can just go, right?

Wylbur (wilbur-wilbur) wrote :

Increasing the boot partition does not solve the underlying problem of old kernels. Is the size large enough now to accommodate enough kernels? If it is, we should tackle the real problem of removing old kernels.

Snoopy (snoopy-u) wrote :

Same problem as indicated above on 14.04 LTS. Why are "old" kernels left on a drive that is too small to begin with? Thank you in advance for fixing it. If fix would allow to expand partition (at least double it) and remove "old" kernels, this would be great!

Cazacu Bogdan (cazacub) on 2015-06-22
information type: Public → Public Security
information type: Public Security → Public
James Donner (jamesadonner) wrote :

Trying this on a Dell Inspiron 15 7547 running 100% Ubuntu (No dual booting). We're unable to upgrade from Ubuntu 14.10 to 15.04 due to this error. We've tried

>sudo apt-get clean
>sudo apt-get autoremove

and nothing has worked. We don't want to get left behind on software updates, so fixing the problem is a high priority.

Hedley Finger (hedley-finger) wrote :

I have this problem but removing old kernels was only partly successful, probably because I used rm and not apt-get method. In fact, if you have a laptop, why use LVM at all? You are scarcely going to wander around with an external hard disk attached to it for the extension to a logical volume.

The installer needs to be a lot smarter about people's level of knowledge re partitions and kernels, and give options for adjusting some of the parameters, e.g. LVM v. normal partitions, size of /boot, guided removal of old kernels, perhaps keeping three consecutive updates (including the current working kernel).

I just went through a torrid attempt to use Gparted (doesn't recognise LVM lv's) and system-config-lvm (which shows the swap and boot partitions at one end of the sda5 red cylinder, and the logical view, which shows them at the OTHER end of the blue cylinder).

I am totally lost in techhead space so am resorting to a new install to "fix" this problem. I chose Ubuntu over Fedora, SuSE and all the rest because it seemed the most novice-friendly, but this ordeal has proved me wrong. 8^)

Remun.J (remun-j66) wrote :

Someone give me the advise to use the LVM option on a fresh Xubuntu 15.04 install so i did. Use one internal drive, a 256GB SSD and a 2TB network drive (small NAS) for external storage. Hope there will be a fix to autoremove those old kernels from /boot partitions before upgrading a new one.

same here, Ubuntu 15.04 64 bit, encrypted [as opposed to the LVM].
Installed about 2015/09/15, and the Update Manager refuses installing already the very first update to the Kernel package(s), i.e., from 3.19.0.28.27
to 3.19.0.30.29.

·

·

·

[I know it's off-topic here, but is there any way round except reinstalling of the whole system? Which btw I know now I have to do manually :) ]

georgebaily (z-george) wrote :

Firstly, me too, this is super annoying. As a rule of thumb any 'feature' that *requires* the user to drop to CLI to put in regular manual commands *to keep the system secure and updated* ... in a desktop / consumer OS... is a bug.

It seems there are separate issues here that could be addressed separately:

1. the auto setup does not intelligently ask/warn about its default of setting up a very small /boot

2. there exists no user-friendly auto cleanup for the things clogging up /boot

3. the /boot does not have any elegant way of automatically (or via a setting) "overflowing" into the (usually abundant) storage space outside of /boot for its old kernel data etc

4. when the "full, cannot update any more" problem occurs, there is no useful feedback or suggestion or wizard from the OS for the user... just essentially a FU message.

5. apparently there is not an "easy" way to resize /boot to allow more crap to accumulate and thus postpone the problem... if there *is* an easy way then see (4) it should be provided then and there to the user; if there is *not* an easy way then this proves the point that this is not a trivial bug.

Addressing at least one of these would help the problem, and all could be worked on.

Ian Weisser (ian-weisser) wrote :

apt-get remove/purge/autoremove is blocked by the (always-failing) kernel install (not enough space on device)

affects: ubiquity (Ubuntu) → apt (Ubuntu)
Ian Weisser (ian-weisser) wrote :

Ubiquity creates space for enough kernels. The system is not managing that space later. Not ubiquity's fault.

These two issues together create the problem on a small /boot partition:
apt: apt-get remove/purge/autoremove is blocked by the (always-failing) kernel install (not enough space on device)
unattended-upgrade: 'autoremove' is disabled by default.

Adding both packages to the bug report. Removing ubiquity.

Workaround - How users can recover from the problem:
- You can use dpkg to manually remove older kernels until apt has enough space to work, or
- You can use dpkg --set selections to un-select the kernel packages for install, then run apt-get autoremove

Workaround - How users can prevent recurrence after successful recovery:
- You can enable autoremoval in /etc/apt/apt.conf.d/50unattended-upgrades , or
- You can mark your wall calendar and manually run apt-get autoremove every few months

summary: - LVM or Encrypted install creates too small /boot partition
+ Kernels not autoremoving, causing out of space error on LVM or Encrypted
+ install with small /boot partition

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in unattended-upgrades (Ubuntu):
status: New → Confirmed

Change summary, because /boot is always to small on "LVM or Encrypted install". The former summary read like it was a user error, while it isn't.

summary: Kernels not autoremoving, causing out of space error on LVM or Encrypted
- install with small /boot partition
+ install
Ian Weisser (ian-weisser) wrote :

Developer of unattended upgrades has submitted a patch:
https://github.com/mvo5/unattended-upgrades/pull/19

Changed in apt (Ubuntu):
status: Confirmed → In Progress
Changed in unattended-upgrades (Ubuntu):
status: Confirmed → In Progress
no longer affects: unattended-upgrades (Ubuntu)
affects: apt (Ubuntu) → unattended-upgrades (Ubuntu)
Achim Behrens (k1l) wrote :

This really needs to be adressed ASAP.

The Support Communites are flooded by angry users with blocked Packagesystems due to a full /boot blocking new kernels to be installed.

In this time, when we have 5TB HDDs and even 500GB SSDs, i really dont know why we still need that small /boot partitions that it runs out of space that fast.
And having no automatic removing old kernels is another issue.

This really spoils the user experience, and non technical users really dont know what to do and dont feel comfortable to delete files in /boot manually. So best is do increase the /boot partition made on install and have a post-install script running that removes old kernels.

Please do not spam bug report with "me too!" or "this bug is awful!"
comments.

The bug report is a living document used by the triagers and developers
to share data, many of them volunteers, and noise makes their job
harder.

A fix has has already been made in the upstream git, and is awaiting
merge. See https://github.com/mvo5/unattended-upgrades/pull/19

There are a number of bugs marked duplicates of this bug that really aren't. The size of /boot is orthogonal to the issue of purging unused kernels. 256 MB gives the ability to install approximately 4 kernels. For some of us that's not enough, and we need to specify a larger /boot. Shortly, I intend to unmark 1465050 as a duplicate, and mark 1511911 as a duplicate of that.

Jarno Suni (jarnos) on 2016-01-11
summary: Kernels not autoremoving, causing out of space error on LVM or Encrypted
- install
+ installation or on any installation, when /boot partition gets full
Jarno Suni (jarnos) wrote :

Ian Weisser, how is Unattended-Upgrade::Remove-New-Unused-Dependencies better than Unattended-Upgrade::Remove-Unused-Dependencies that is already available? Why not just use a specific script to remove old kernels instead of using apt-get autoremove?

daniel smith (opened-to) wrote :

this is affecting me too.

Filesystem Size Used Avail Use% Mounted on
udev 981M 4.0K 981M 1% /dev
tmpfs 200M 1.1M 199M 1% /run
/dev/mapper/ubuntu--vg-root 72G 7.4G 61G 11% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 997M 404K 996M 1% /run/shm
none 100M 52K 100M 1% /run/user
/dev/sda1 236M 211M 14M 95% /boot
/home/rebecca/.Private 72G 7.4G 61G 11% /home/rebecca

i thought that it was because my drive was full. i think its only maybe 40 gig?
thanks

Will the fix, that has been merged in the meantime make it into 14.04 LTS or do we have to wait for some future Ubuntu release?

Jarno Suni (jarnos) wrote :

Fixing the issue using unattended upgrades in 14.04 LTS requires fixing Bug #1492709, too.

Jarno Suni (jarnos) wrote :

And it still would require manually removing the kernels installed before the fix is installed.

Ian Weisser (ian-weisser) wrote :
Changed in unattended-upgrades (Ubuntu):
status: In Progress → Fix Committed
Ian Weisser (ian-weisser) wrote :

Fix included in upstream release: unattended-upgrades 0.89 30 JAN 2016
Uploaded to Ubuntu 16.04 on 02 FEB 2016.

   * Add `Unattended-Upgrade::Remove-New-Unused-Dependencies` that
     defaults to "yes". This ensures that older kernel
     get automatically cleaned up and /boot will not overflow.
     LP: #1267059

Marking this bug as a duplicate of LP: #1267059 so it gets closed automatically when appropriate.

Jarno Suni (jarnos) wrote :

LP: #1267059 is about automatic removing fails even if
Unattended-Upgrade::Remove-Unused-Dependencies "true";
It was fixed already in unattended-upgrades - 0.76ubuntu1.2. There is an error in the changelog of unattended-upgrades. This is not a duplicate of bug #1267059. This bug is about making the autoremoval of excessive the default behavior.

Jarno Suni (jarnos) wrote :

Can someone attach the current default /etc/apt/apt.conf.d/50unattended-upgrades of ?Ubuntu 16.04 (clean install)? Is there "Remove-New-Unused-Dependencies" advertised in comment #36? Is there line
Unattended-Upgrade::Remove-Unused-Dependencies "true";
there and is it uncommented? What is in the Unattended-Upgrade::Allowed-Origins block? Is unattended-upgrades enabled by default: What does Updates-tab show in Software & Updates (i.e. software-properties-gtk), particularly items "When there are * updates". What does "sudo dpkg-reconfigure unattended-upgrades" show?

I upgraded from 15.10 i.e. not clean install, and there is no "Remove-New-Unused-Dependencies" in /etc/apt/apt.conf.d/50unattended-upgrades.

Jarno Suni (jarnos) wrote :

50unattended-upgrades does not contain mention for Unattended-Upgrade::Remove-New-Unused-Dependencies, but the default value is true. Automatic installing of security updates is enabled by default in Xenial.

Jarno Suni (jarnos) wrote :

Fixed in version 0.90

Changed in unattended-upgrades (Ubuntu):
status: Fix Committed → Fix Released
Jarno Suni (jarnos) on 2016-06-07
description: updated
24 comments hidden view all 104 comments
Jarno Suni (jarnos) wrote :

Ta, which version of Ubuntu do you mean? Did you try the workaround mentioned in the bug description?

Jarno Suni (jarnos) wrote :

Ta, oh, it seems you have tried to apply the workaround earlier. If you removed extra kernels then and setup Unattended Upgrades to install and remove kernels automatically, /boot folder should have some space, so I don't understand why it was full again.

Jarno Suni (jarnos) wrote :

Ta, can you upload your 10periodic, 20auto-upgrades and 50unattended-upgrades from /etc/apt/apt.conf.d? and some log file from /var/log/unattended-upgrades that tells about kernel upgrades?

Changed in unattended-upgrades (Ubuntu):
importance: Undecided → High
tags: added: precise trusty xenial
Mathew Hodson (mathew-hodson) wrote :

This was fixed in Xenial. The default is now to remove newly unused dependencies after every unattended upgrade.

---
unattended-upgrades (0.89) unstable; urgency=medium

  [ Michael Vogt ]
  * Add `Unattended-Upgrade::Remove-New-Unused-Dependencies` that
    defaults to "yes". This ensures that older kernel
    get automatically cleaned up and /boot will not overflow.
    LP: #1267059
  * Remove downloaded deb packages after successful installs.
    This can be controlled via the option:
    `Unattended-Upgrade::Keep-Debs-After-Install`
    (Closes: #809428)
  * Only remove debs in the cache dir

  [ Steffen Köhler ]
  * Allow to configure sender email via `Unattended-Upgrade::Sender`

  [ Antti Riikonen ]
  * Minor typo fix in comment

  [ Cleto Martin ]
  * bugfix: non-ascii chars in dpkg log file crashes unattended upgrades
    Closes: #812857

  [ James Valleroy ]
  * Set debconf value for auto updates based on current configuration.

  [ Alexandre Detiste ]
  * update French transalation

 -- Michael Vogt <email address hidden> Sat, 30 Jan 2016 16:04:52 +0100

tags: removed: xenial
Changed in unattended-upgrades (Ubuntu):
importance: High → Wishlist
Mathew Hodson (mathew-hodson) wrote :

I don't think backporting the new defaults to Precise and Trusty would be right for an SRU.

On Precise and Trusty, you can set Unattended-Upgrade::Remove-Unused-Dependencies "true";

This is false by default. Setting it to true will remove *all* unused dependencies after every unattended upgrade.

Jarno Suni (jarnos) wrote :

Unattended upgrades does not remove old kernels that were installed by other means than unattended upgrades even if "Unattended-Upgrade::Remove-Unused-Dependencies "true";" is set in Trusty (and maybe in Precise) Bug #1492709

Jarno Suni (jarnos) wrote :

I think this `Unattended-Upgrade::Remove-New-Unused-Dependencies` is not well tested yet. I have not noticed if it has removed any kernel. Also, I think it will not remove a kernel, if you happen to install a new kernel using Software Updater before Unattended Upgrades operates.

Ian Weisser (ian-weisser) wrote :

There seems a lot of confusion in this bug report about which pieces of the system work together to cause this problem, and which pieces should be responsible for fixing it.

Marking kernels as eligible for autoremoval is done by apt. The apt package provides /etc/kernel/postinst.d/apt-auto-removal, which sets the rules for marking kernel packages eligible for autoremoval. If you discover a system that keeps more than three kernels (and isn't already broken), run 'apt autoremove' manually. If the system still has more than three kernels, then see if you can figure out why and then file a bug against Apt.

But apt only marks. Apt doesn't actually pull the trigger and run an autoremove.

Compounding the problem is another issue with apt: Apt in 16.04 and earlier queues autoremoval instead of running it first (or last...or scheduled). This is what makes broken systems harder to fix. Out-of-space hits before autoremove.

Unattended-Upgrades prior to 16.04, if the option is enabled, merely runs 'apt autoremove' via aptdaemon. U-U does not check nor care what packages need to be removed. U-U does not check nor care if the system is LVM/Encrypted or has a small separate /boot partition. None of that is part of U-U's job.

U-U developers in 16.04 addressed this problem (even though it's arguably not U-U's problem) by checking if packages are already marked eligible-for-autoremove at the beginning of the run. If so, it autoremoves them before doing anything else. Again, U-U doesn't care about which packages happen to be marked, or wheter the system has a small /boot partition.

In 16.04, the problem is undeniably fixed. The additional functionality of U-U has prevented stale kernels from accumulating on all my test systems. However, developers have understandably shown little interest in backporting that particular fix.

And there are other options to integrate the fix better into apt itself instead of U-U. For example, aptdaemon does seem to have queue management tools - it should be possible to move an 'autoremove' to the head of queue (incidentally making fixes much easier). Or we can add another kernel hook file to /etc/kernel/postinst.d/ to trigger an autoremove upon kernel installs (instead of U-U's daily check). Or we can add a startup/cron.daily service that simply checks for existence of a small /boot partition and runs autoremove before (or after) the daily apt update. Or we can have aptdaemon catch out-of-space errors (instead of simply passing them), run autoremove, and then try the install again.

Almost none of these possible solutions would be implemented in Unattended Upgrades.

Brian Candler (b-candler) wrote :

I have two Precise (12.04) servers with

Unattended-Upgrade::Remove-Unused-Dependencies "true";

in 50unattended-upgrades. One of them cleans up its kernels and only keeps the last two; one of them accumulates kernels over time, and I occasionally get alerts about /boot filling up.

On the latter machine, "apt-get autoremove --purge" doesn't remove them. I end up removing the packages individually; the script "purge-old-kernels" referred to above doesn't seem to exist.

The main difference I can see is that the latter machine has linux-image-generic-lts-trusty, whereas the first machine has linux-image-server.

Both machines have in /etc/apt/apt.conf.d/05aptitude:

aptitude::Keep-Unused-Pattern "^linux-image.*$ | ^linux-restricted-modules.*$ | ^linux-ubuntu-modules.*$";

So I don't think that's it. I suppose there won't be too many kernel updates for Precise before it goes end of life now though...

Jarno Suni (jarnos) wrote :

Brian Candler, check if you have same repositories enabled in both servers. You should get purge-old-kernels, if you add byobu PPA repository.

Jarno Suni (jarnos) wrote :

Ian Weisser, I think in 16.04, in contrary to what you claim, packages are not autoremoved before doing anything else. https://github.com/mvo5/unattended-upgrades tells:
"Unattended-Upgrade::Remove-New-Unused-Dependencies - boolean (default:True)

Remove any new unused dependencies after the upgrade finished."

Anyway, in my 16.04 system, where I have the setting enabled (as default) and unattended-upgrades enabled (via Software & Updates), I currently have 6 kernels installed, so I think it is not doing its job well.

Ian Weisser (ian-weisser) wrote :

Jarno Suni,

Okay, I stand corrected on that statement.

Please explain how you determined that Unattended Upgrades is the culprit, instead kernel header packages or user changes to apt-marking or other possible causes for the symptom you describe.

With the information so far, I cannot duplicate your issue on my test system. How can I do so?

Jarno Suni (jarnos) wrote :

I guess the reason is I have installed some kernels using Software Updater even if Unattended Upgrades is set to install security updates. I guess I am not the only one using update-notifier and update-manager to install updates.

Jarno Suni,

That possible cause seems irrelevant to Unattended-Upgrades. Any
package installed by any front-end to apt will be upgraded. U-U doesn't
know what packages are installed, nor care. U-U simply tells aptdaemon
to upgrade packages from the repositories authorized by the user. U-U
does not duplicate nor replace Software Updater, and both can safely be
used.

Try running 'apt autoremove' (16.04) or 'apt-get autoremove'
(12.04/14.04).

If those kernels are not removed, then your problem is the apt-marking,
not U-U. There are several ways to prevent automatically marking kernel
packages as 'auto' - the most common is by installing kernel header
packages.

If those kernels are removed, then check the U-U logs in /var/log to
ensure that U-U is really operating.

A bug report is an awful place to try to troubleshoot in this way. If
you open a thread on UbuntuForums, we may be able help you better.

Jarno Suni (jarnos) wrote :

Ian Weisser, the issue is covered by the title of this bug report, anyway. If 'Unattended-Upgrade::Remove-Unused-Dependencies "true";' is used, U-U removes extra kernels, even if new kernels are installed by Software Updater in 16.04. In 14.04 (and in 12.04 I suppose) it does not work due to bug Bug #1439769. So running 'apt-get autoremove' in 14.04/12.04 may not have the desired effect. Anyway, the point of this bug report was that old kernels should be autoremoving so that administator does not have to be concerned of /boot getting full, so I think another package should be added as being affected, too.

How does installing kernel header packages (which do normally get installed anyway) prevent marking kernel packages as 'auto'? I do not see that happening in 16.04 at least. linux-header packages do not depend on linux-image packages besides linux-header packages may itself be marked 'auto'.

Ian Weisser (ian-weisser) wrote :

Jarno Suni,

This bug report ran it's course, and 'Fix Released' is another way of
saying 'Closed.' You can discuss it, but no developer is likely to read
it...to them, it's a closed issue. It's fixed, and won't be unfixed.
The patch is released, and won't be reverted. The normal workflow is to
file a new bug, not to reopen or discuss a closed bug.

Bug 1357093 is 'Fix Released' because I duplicated the original
reporter's problems, triaged the bug, and forwarded enough information
to a developer who agreed with my triage and decided to fix it. Upon
testing the fix (repeatedly), my duplication of the original reporter's
problem is really fixed.

If you have a triage-able problem with the unattended-upgrades package,
please open a new bug report and provide enough information so I can
triage it. I cannot generally triage a problem that I cannot duplicate.

If all you know is 'it should work but does not work', then please jump
into a support forum (like Ubuntuforums) and let the experts there help
you figure out how to duplicate or triage it. Similar symptoms may come
from very different causes. Having an unresolved problem is really,
really frustrating. Been there.

I'm quite happy to discuss kernel header dependencies and go into much
greater detail...but doing so in this bug report would be rather
counterproductive. The issue is closed, and most of the questions seem
unrelated to the original bug.

Jarno Suni (jarnos) wrote :

This bug is 'Fix Released' because I marked it as such. Maybe too soon as it is irreversible.

Ok, if someone writes new bug report about kernels not autoremoving, please link it here.

Ian Weisser, you may e.g contact me privately, if you want to tell me about the kernel header packages thing.

Jarno Suni (jarnos) wrote :

Bug #1624644 claims U-U does not work well with Software Updater a.k.a. update-manager. Can you confirm?

evendough people talk about not adding more comments to this thread.

i had the problem, then they/you/here they said solution is upgrade to 16.04

- i solved the problem manually und than upgraded to 16.04

- now some months later i wanted to update && upgrade my server and boot is full again. - and i had to solve it manually again.

so for my experience > fix didn't work.

what i did is:
- nothing
- just have the unatended auto updater install security updates. (configured as in 14.04) super default basic.

i now loocket in to the "50unattended-upgrades" file and

// Do automatic removal of new unused dependencies after the upgrade
// (equivalent to apt-get autoremove)
// Unattended-Upgrade::Remove-Unused-Dependencies "false";

is commented..
Would i need to change that? or is that the unrelated one?

Jarno Suni (jarnos) wrote :

fksdlofioenvdfsdji,
How did you "want to update && upgrade your server"?
How many kernels can your /boot partition hold? (What is the size of your /boot partition?)

Contents of "50unattended-upgrades" is misleading, as there is no mention of the new configuration variable Unattended-Upgrade::Remove-New-Unused-Dependencies which defaults to "true" in the file (in Ubuntu 16.04).

Replacing line
// Unattended-Upgrade::Remove-Unused-Dependencies "false";
by
Unattended-Upgrade::Remove-Unused-Dependencies "true";
may help you, if you don't mind that it removes all packages that 'sudo apt-get autoremove' would.

Does anyone have this problem in Linux Mint 18? "Unattended-Upgrade::Remove-New-Unused-Dependencies" also does not exist for me but the package was last updated a month ago, which leads me to believe the configuration file "50unattended-upgrades" wasn't "replaced" with the new version. Also, the package has other issues including not fetching from the correct repos as it's a stock Ubuntu config.

C Filorux (breakfast) wrote :

The fix is failing in xenial: I have to repeatedly clean up after the installer, when it complains that it cannot do its work due to disk space on the tiny /boot partition. "My disk is full again!"

The bug remains: The automatic update is not considered an unattended update, since the user is there attending it via an interactive application (tell me I'm wrong).

Ian Weisser (ian-weisser) wrote :

Another (complimentary) solution for accumulating kernels in an LVM environment: http://blog.surgut.co.uk/2016/11/boot-less-lvm-rootfs-in-zesty.html

The fix to this bug prevents kernels from accumulating.
That complimentary solution eliminates the tiny /boot partition, eliminating the space problem on LVM desktops and laptops...though not on other space-constrained devices.

My updated u-u on Ubuntu 16.04 and 16.10 is working just great, and correctly preventing kernels from accumulating. However, we have seen similar-looking apt problems in the support forums: A few systems to continue to run out of /boot space. Upon investigation, these are usually due to unconfigured/half-configured packages or user confusion with the settings.

If your unattended-upgrade is not working properly, and your system continues to run out of /boot space, please seek help in a support forum before posting permanently to this CLOSED bug report.

1 comments hidden view all 104 comments
Mossroy (mossroy) wrote :

Ian, could you please give the URLs where to find the "similar-looking apt problems in the support forums" you're mentioning?
I think many people here (including me) would be interested in what are the correct settings you mention.

Ian Weisser (ian-weisser) wrote :

Happy to discuss and help...in the support forums.
This CLOSED bug report is not the appropriate location to help people troubleshoot.

Mitch Singler (msingler) wrote :

Seems to be a design flaw rather than a bug.

      From: Ian Weisser <email address hidden>
 To: <email address hidden>
 Sent: Tuesday, November 22, 2016 9:26 PM
 Subject: [Bug 1357093] Re: Kernels not autoremoving, causing out of space error on LVM or Encrypted installation or on any installation, when /boot partition gets full

Happy to discuss and help...in the support forums.
This CLOSED bug report is not the appropriate location to help people troubleshoot.

--
You received this bug notification because you are subscribed to a
duplicate bug report (1183692).
https://bugs.launchpad.net/bugs/1357093

Title:
  Kernels not autoremoving, causing out of space error on LVM or
  Encrypted installation or on any installation, when /boot partition
  gets full

Status in unattended-upgrades package in Ubuntu:
  Fix Released

Bug description:
  Currently if one chooses to use LVM or encrypted install, a /boot
  partition is created of 236Mb

  Once kernel updates start being released this partition soon fills
  until people are left unable to upgrade.

  While you and I might know that we need to watch partition space, many
  of the people we have installing think that a windows disk is a disk
  and not a partition, education is probably the key - but in the
  meantime support venues keep needing to deal with the fact the
  partition is too small and/or old kernels are not purged as new ones
  install.

  For workaround and sytem repair, see
  https://help.ubuntu.com/community/Lubuntu/Documentation/RemoveOldKernels

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/unattended-upgrades/+bug/1357093/+subscriptions

Mossroy (mossroy) wrote :

For those still having this issue on Ubuntu 16.04 : please check that you did not disable the automatic installation of updates.
It might be the reason why the old kernels were not removed for me (not 100% yet). See discussion in https://ubuntuforums.org/showthread.php?t=2344232
With a GUI, it's in System settings->Software and updates->Updates tab : "When there are security updates" should be set to "Download and install automatically" (its default value).
(please comment on the forum thread)

Henry (hvanmegen) wrote :

I can't even run `apt-get autoremove` because it starts filling up my /boot with old kernels until it runs out of space.. who the hell designed this madness?

Henry (hvanmegen) wrote :

Ok, I had to manually remove the files as they were added.. after that, it worked as intended.

Funny thing is, all solutions provide here say that you need to enable the removal of unused dependencies by setting this:

`Unattended-Upgrade::Remove-Unused-Dependencies "true"`

Problem is.. I already had this set in my `/etc/apt/apt.conf.d/50unattended-upgrades`..

This is broken, design or not.. it IS a bug.. how can we fix this permanently? I have servers running unattended upgrades that have overflowing /boot partitions which is annoying AF.

Jarno Suni (jarnos) wrote :

Henry, do you use Ubuntu Trusty, and install kernels manually? See comment #79.

Bryce Nesbitt (bryce2) wrote :

My elderly parents are on Ubuntu 16.04, with updates set to 'Download and install automatically'.
I must manually prune the Kernel images every so often, or the system's underwear gets all in a twist, and "apt-get autoremove' won't even work.

Let's be clear: THIS IS NOT FIXED as of 16.04 LTS.

spike speigel (frail-knight) wrote :

This might be aggravated by this DKMS bug which was found in v2.2.0.3, which incidentally is used in Ubuntu 16.04 LTS, and maybe other versions. It doesn't remove old initrd files in /boot *which lead to full /boot issues and subsequent update problems*

I manually removed a bunch of old initrd images the other day.

http://askubuntu.com/questions/616512/purging-old-kernels-fails-to-remove-old-initrd-files

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717584

James Duvall (jamesduvall) wrote :

running 16.10, I am also running in to the problem that autoremove fails to remove old initrd files. This was greatly exacerbated by bug #798414 (https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/798414). With an encrypted file system and a small boot partition, I only have space for 3 kernels. With the pace of new kernel builds, it doesn't take long to run out of space. Now that I understand the problem, it is not such a big deal, but it was very annoying to have unhelpful initramfs errors nearly every time I boot up.

Jackalux (heartwoodjack) wrote :

I don't have hardware that I imagine will run 16.xx so need this addressed on older LTS versions.

My output for...
dpkg -l 'linux-*' | sed '/^ii/!d;/'"$(uname -r | sed "s/\(.*\)-\([^0-9]\+\)/\1/")"'/d;s/^[^ ]* [^ ]* \([^ ]*\).*/\1/;/[0-9]/!d'
 ... was 120 lines long! I had to teach myself scripting to delete them all manually. I only discovered the error after a different Linux machine I created (with a separate /boot) ran into the bug.

Deleting this junk freed 10GB of total disk space.

This is too much to ask of novice users and will stop Linux being more widely adopted. I'm not a complete Noob but this was annoying to deal with... it will be impossible for others.

Andreas Lindhé (lindhe) wrote :

I hate to be that guy that complains without contributing in a constructive manner, but I just need to say that it's completely unreasonable that my mom and friends I recommend Ubuntu to gets this problem. They should never have to be concerned with what a kernel image is or why the boot partition is full.

Jarno Suni (jarnos) wrote :

Jackalux, oh I see you have found the popular one-liner script for kernel removing. It is too complicated for what it does and too simple for what it should possibly do.

I have written a script to ease kernel removal:
https://www.bountysource.com/issues/38300038-feature-request-the-command-should-work-like-this
You (and other complainers) could post a small bounty to fund the work. Alternatively I think we could form an association to fund the work and other stuff that novice Ubuntu users may need provided it could attract enough paying members.

Andreas Lindhé, you have a point. I don't know who decides it, but I suppose the fix to the bug could be backported or SRUed to even older LTS releases. But it seems that the fix does not work even in 16.04 (Bug #1675079); that applies only to the new Unattended-Upgrade::Remove-New-Unused-Dependencies setting in unattended-upgrades, though, so if setting Unattended-Upgrade::Remove-Unused-Dependencies "true" suits to you, there is a fix available. Alternatively, e.g. my script could be run from a cron job or (in 16.04 or later) from a systemv timer to handle automatic kernel removal.

Jarno Suni (jarnos) wrote :

Correction: I meant systemd, not systemv in the previous comment.

Displaying first 40 and last 40 comments. View all 104 comments or add a comment.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.