linux-headers will eat your inodes on LTS.

Bug #1089195 reported by Pablo Piaggio
This bug affects 37 people
Affects Status Importance Assigned to Milestone
update-manager (Ubuntu)

Bug Description

Hello all,

Both linux-image-* and linux-headers-* are installed every time you upgrade the kernel. However, they are never removed by any maintenance process.

Every linux-headers-*-generic-pae package has approx. 6,700 files on it. Regular headers packages have even more files: around 11,700 files each.

Although these packages' files won't occupy much space, after some years (think LTS installation), they will "eat" all the inodes on your root partition.

The first effect you'll encounter will be the you are unable to upgrade your system.

This is a situation that will affect all users, however it will be a greater problem regular non-technical user.

There's no simple, high-level tool to solve this problem.

Case Study
I have a 2-year-and-8-months old 10.04 installation. I have a ~10GB root partition, which is double of the minimum recommend (5GB).

    $ df -h --type=ext4
    Filesystem Size Used Avail Use% Mounted on
    /dev/sda5 9.4G 6.2G 2.8G 70% /
    /dev/sda6 94G 45G 45G 51% /home

It looks that I had plenty of space left to upgrade, but I got this error while upgrading:

    unable to create `/usr/src/linux-headers-2.6.32-45/arch/s390/include/asm/nmi.h.dpkg-new'
    (while processing `./usr/src/linux-headers-2.6.32-45/arch/s390/include/asm/nmi.h'): No space left on device

It was because of I was running out of inodes:

    $ df -i /
    Filesystem Inodes IUsed IFree IUse% Mounted on
    /dev/sda5 625856 618015 7841 99% /

Why? Because the huge amount of linux-headers files:

    $ find /usr/src -type f | wc -l

That is more than three hundred and fifty thousand files!

These are the packages that I removed:

Purg linux-headers-2.6.32-21 [2.6.32-21.32]
Purg linux-headers-2.6.32-22-generic-pae [2.6.32-22.36]
Purg linux-headers-2.6.32-22 [2.6.32-22.36]
Purg linux-headers-2.6.32-23-generic-pae [2.6.32-23.37]
Purg linux-headers-2.6.32-23 [2.6.32-23.37]
Purg linux-headers-2.6.32-24-generic-pae [2.6.32-24.43]
Purg linux-headers-2.6.32-24 [2.6.32-24.43]
Purg linux-headers-2.6.32-25-generic-pae [2.6.32-25.45]
Purg linux-headers-2.6.32-25 [2.6.32-25.45]
Purg linux-headers-2.6.32-26-generic-pae [2.6.32-26.48]
Purg linux-headers-2.6.32-26 [2.6.32-26.48]
Purg linux-headers-2.6.32-27-generic-pae [2.6.32-27.49]
Purg linux-headers-2.6.32-27 [2.6.32-27.49]
Purg linux-headers-2.6.32-28-generic-pae [2.6.32-28.55]
Purg linux-headers-2.6.32-28 [2.6.32-28.55]
Purg linux-headers-2.6.32-29-generic-pae [2.6.32-29.58]
Purg linux-headers-2.6.32-29 [2.6.32-29.58]
Purg linux-headers-2.6.32-30-generic-pae [2.6.32-30.59]
Purg linux-headers-2.6.32-30 [2.6.32-30.59]
Purg linux-headers-2.6.32-31-generic-pae [2.6.32-31.61]
Purg linux-headers-2.6.32-31 [2.6.32-31.61]
Purg linux-headers-2.6.32-32-generic-pae [2.6.32-32.62]
Purg linux-headers-2.6.32-32 [2.6.32-32.62]
Purg linux-headers-2.6.32-33-generic-pae [2.6.32-33.72]
Purg linux-headers-2.6.32-33 [2.6.32-33.72]
Purg linux-headers-2.6.32-34-generic-pae [2.6.32-34.77]
Purg linux-headers-2.6.32-34 [2.6.32-34.77]
Purg linux-headers-2.6.32-35-generic-pae [2.6.32-35.78]
Purg linux-headers-2.6.32-35 [2.6.32-35.78]
Purg linux-headers-2.6.32-36-generic-pae [2.6.32-36.79]
Purg linux-headers-2.6.32-36 [2.6.32-36.79]
Purg linux-headers-2.6.32-37-generic-pae [2.6.32-37.81]
Purg linux-headers-2.6.32-37 [2.6.32-37.81]
Purg linux-headers-2.6.32-38-generic-pae [2.6.32-38.83]
Purg linux-headers-2.6.32-38 [2.6.32-38.83]
Purg linux-headers-2.6.32-39-generic-pae [2.6.32-39.86]
Purg linux-headers-2.6.32-39 [2.6.32-39.86]
Purg linux-headers-2.6.32-40-generic-pae [2.6.32-40.87]
Purg linux-headers-2.6.32-40 [2.6.32-40.87]
Purg linux-headers-2.6.32-41-generic-pae [2.6.32-41.94]
Purg linux-headers-2.6.32-41 [2.6.32-41.94]
Purg linux-headers-2.6.32-42-generic-pae [2.6.32-42.96]
Purg linux-headers-2.6.32-42 [2.6.32-42.96]
Purg linux-headers-2.6.32-43-generic-pae [2.6.32-43.97]
Purg linux-headers-2.6.32-43 [2.6.32-43.97]
Purg linux-headers-generic-pae []
Purg linux-image-2.6.32-42-generic-pae [2.6.32-42.96]
Purg linux-image-2.6.32-43-generic-pae [2.6.32-43.97]

Then, problem solved:

    $ find /usr/src/ -type f | wc -l

    $ ls /usr/src/
    linux-headers-2.6.32-44 linux-headers-2.6.32-45-generic-pae
    linux-headers-2.6.32-44-generic-pae nvidia-current-195.36.24
    linux-headers-2.6.32-45 virtualbox-ose-3.1.6

    $ df -i --type=ext4
    Filesystem Inodes IUsed IFree IUse% Mounted on
    /dev/sda5 625856 192891 432965 31% /
    /dev/sda6 6225920 95025 6130895 2% /home

Revision history for this message
Pablo Piaggio (papibe) wrote :

This just happened to me on another 10.04.4 server installation. All while trying to upgrade kernel versions. This is an LTS version, and it has support until April 2015, however the old headers are still accumulating and eating inodes.

Please, I beg of you, take notice of this issue.

    $ uname -a
    Linux reinhardt 2.6.32-50-generic-pae #112-Ubuntu SMP Tue Jul 9 20:44:31 UTC 2013 i686 GNU/Linux

    $ lsb_release -a
    No LSB modules are available.
    Distributor ID: Ubuntu
    Description: Ubuntu 10.04.4 LTS
    Release: 10.04
    Codename: lucid

    $ df -i /
    Filesystem Inodes IUsed IFree IUse% Mounted on
    /dev/sda1 305216 302123 3093 99% /

   $ ls -CF /usr/src

   $ find /usr/src/ -type f | wc -l

Revision history for this message
NickW (e-launchpad-wu-lee-noodlefactory-co-uk) wrote :

I also have just run into this problem on an installation of Ubuntu Lucid, owned by a non-technical user.

The machine had become unusable - many programs could not run, including the package manager. Symptoms were totally baffling for the user: although some errors appear claiming that there is not enough space, a check of the filesystem capacity using GUI tools shows there is in fact plenty of space.

Indeed, this foxed me for a while, as even when it was evident that the limit was the number of inodes rather than storage space, there seems to be no easy way to find what is using up inodes except by running shell commands which count files, a slow and painful process, especially when directing a user over the phone.

The issue: about 20 kernel package upgrades using up all the available inodes on the root filesystem. Uninstalling these freed 55% of a total of 650000 inodes. Entirely simple, and with a bit of polish, could be made totally avoidable with a suitable option. There's no need to keep so many kernel packages around.

Removing old kernels does require more effort than it would appear to deserve (list the packages, grep, pipe into apt-get remove). It would make sense to me if the automatic updates manager could be told to handle removing kernels too.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in update-manager (Ubuntu):
status: New → Confirmed
Revision history for this message
Christian Dannie Storgaard (cybolic) wrote :

I'd also suggest that the system warns a user when a filesystem is close to running out of inodes, in a similar fashion to the low disk space warning.

Revision history for this message
John Tucker (jonti) wrote :

I've a Dell Inspiron mini which came with Ubuntu installed. It's running 12.04 lts and does very little apart from tv listings and monitoring email (it's too small for regular use). Yet it ran out of inodes as shown by df -i when the drive was about half-full according to df -h.

This borked my ability to update packages and resulted in aptitude reporting broken packages after an update attempt. Worse, it became impossible to login to a graphical desktop. To recover, I needed to boot into text mode by amending the grub boot-up commands and deleting /tmp to free up sufficient inodes to allow the graphical desktop to load ok. Then I used synaptic to delete the old linux-headers. The result was inode usage dropped to a more reasonable 54%.

So yeah, untold numbers of Ubuntu installations are in danger of mysteriously stopping working because ubuntu does not tidy up after itself when installing new kernels.

Revision history for this message
Gregorio Bastardo (gregorio-bastardo) wrote :

I just ran into this issue with Ubuntu 12.04, after using it for more than a year.

I have to say that it's very embarrassing to spend some time to find the root cause of an update manager crash due to full inode because of ancient linux headers. These files should be removed by the system on regular maintenance or at least suggested to the administrator when critical inode capacity detected.

Worst, this issue has been reported more than a year ago. Since it's unresolved, I consider Ubuntu is still only for technical users.

Revision history for this message
pabouk (pabouk) wrote :

This is really annoying bug. Even experienced user who never encountered a problem with consumed inodes will spend a lot of time finding the cause of the problem. The error message "No space left on device" does not suggest that the problem is with inodes.

The update process should certainly remove very old kernels automatically by default or show an explanatory warning.

There are many examples of users which faced this problem:

See also related Bug #690911 Installation without formatting fails to remove old kernels

Attempts (certainly imperfect) to remove old kernels automatically:

Revision history for this message
JohnWashington (ubuntu-johnwash) wrote :

Yes, I'm an experienced user and I too have spent a lot of time tracking down why a 12.04 LTS update failed. The system has separate / and /home partitions, and the / partition is 15GB, yet "No space left on device" was suggested as the likely cause. There are many dozens of old kernels.

At the very least, the error message "No space left on device" should be augmented with "or inode shortage". But this will not solve the problem for non-technical users.

12.04 LTS is supposed to be supported for more than 2 years yet.

Revision history for this message
Nils Toedtmann (m-launchpad-net-mail-nils-toedtmann-net) wrote :

This issue made an upgrade fail in the middle which left my system (12.04.5 LTS) with broken dependancies that are not trivial to solve: "apt-get -f install" fails due to lack of inodes. "apt-get autoremove" refuses to run due to broken deps, and so does "apt-get remove -f $SOME_OLD_KERNEL_PACKGES".

In my case, the depending packages were luckily meta-packages (linux-headers-server linux-server), so i could "remove -f" them, then "autoremove" worked again a threw away lots of old header packages, then re-"install linux-headers-server linux-server" and finally "upgrade".

Not sure how an unexperienced user is supposed to go about such a situation.

Revision history for this message
Nils Toedtmann (m-launchpad-net-mail-nils-toedtmann-net) wrote :

Note that automatic updates (e.g. "unattended-upgrades") will even more likely bring you into this situation. And because of bug #1267059, even then you set 'Unattended-Upgrade::Remove-Unused-Dependencies true'. Not good for a LTS.

Revision history for this message
fermulator (fermulator) wrote :

How can we increase the severity/importance here to get attention? When this happens, usually we can get out of it, but I ran into this annoying situation of conflicting packages. I couldn't do the usual, because:
fermulator@fermmy-server:/dev/disk/by-id$ sudo apt-get install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following extra packages will be installed:
  linux-generic-pae linux-headers-3.2.0-84 linux-headers-3.2.0-84-generic-pae linux-headers-generic-pae linux-image-3.2.0-84-generic-pae linux-image-generic-pae
Suggested packages:
  fdutils linux-doc-3.2.0 linux-source-3.2.0 linux-tools
The following NEW packages will be installed:
  linux-headers-3.2.0-84 linux-headers-3.2.0-84-generic-pae linux-image-3.2.0-84-generic-pae
The following packages will be upgraded:
  linux-generic-pae linux-headers-generic-pae linux-image-generic-pae
3 upgraded, 3 newly installed, 0 to remove and 160 not upgraded.
7 not fully installed or removed.
Need to get 0 B/51.1 MB of archives.
After this operation, 182 MB of additional disk space will be used.
Do you want to continue [Y/n]?
(Reading database ... 700441 files and directories currently installed.)
Unpacking linux-image-3.2.0-84-generic-pae (from .../linux-image-3.2.0-84-generic-pae_3.2.0-84.121_i386.deb) ...
Unpacking linux-headers-3.2.0-84 (from .../linux-headers-3.2.0-84_3.2.0-84.121_all.deb) ...
Unpacking linux-headers-3.2.0-84-generic-pae (from .../linux-headers-3.2.0-84-generic-pae_3.2.0-84.121_i386.deb) ...
dpkg: error processing /var/cache/apt/archives/linux-headers-3.2.0-84-generic-pae_3.2.0-84.121_i386.deb (--unpack):
 unable to create `/usr/src/linux-headers-3.2.0-84-generic-pae/include/config/ir/imon.h.dpkg-new' (while processing `./usr/src/linux-headers-3.2.0-84-generic-pae/include/config/ir/imon.h'): No space left on device
No apport report written because MaxReports is reached already
                                                              dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)
Errors were encountered while processing:
E: Sub-process /usr/bin/dpkg returned an error code (1)


Revision history for this message
fermulator (fermulator) wrote :

(had to manually temporarily move stuff to a different partition from /usr/src)

Revision history for this message
B. (b-deactivatedaccount-deactivatedaccount) wrote :

My workaround:
dpkg --get-selections | grep 'linux-'
dpkg --get-selections | awk '/linux-(headers|image)-[0-9]\./ { print $1 }' | grep -v "$(uname -r | sed -e 's/-generic//')" | sort -r -V -t- -k3 | tail -n+4 | xargs -r apt-get -qq -y purge
apt-get -qq -y install linux-{headers,image}-$(uname -r)
dpkg --get-selections | grep 'linux-'

# (optional)
# apt-get -qq -y install linux-{headers,image}-generic-lts-utopic 2>/dev/null || apt-get -qq -y install linux-{headers,image}-generic-lts-trusty

Until a backport is available for LTS 14.04/12.04 see

Revision history for this message
KFlash (kuv02) wrote :

This bug is quite annoying. I ran into it, too. And I tend to be more one of the "more regular users" that doesn't see an "easy high lever approach to solve this", as the reporter said.

Please build a cleanup script into the upgrade procedure that purges all kernel files and headers of versions more then 2 to 3 versions back. Why should you keep them anyway?

Revision history for this message
Charles (slivkoff) wrote :

This just hit me on 14.04 LTS.

Any ETA on a fix?

Revision history for this message
algal (alexis) wrote :

I have just encountered this issue on an Ubuntu 14.04 LTS instance. It was setup in early 2014, and has had only a light workload serving an API backend for a medical education system in use by many doctors.

Ironically, it seems to be _because_ I enabled unattended-upgrades that the system now cannot update itself.

And I not sure how to repair this situation without interrupting service, since many package management commands depend on first doing apt-get update, which is what I cannot do. And I don't think I understand the system well enough to be sure that if I erase something manually it is not critical.

If anyone has advice on this, I'd appreciate, even if only pointers to books on more esoteric aspects of Ubuntu package management.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.4 LTS
Release: 12.04
Codename: precise

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 5.9G 1.7G 78% /
udev 288M 12K 287M 1% /dev
tmpfs 60M 196K 59M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 296M 0 296M 0% /run/shm

$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/xvda1 524288 518210 6078 99% /
udev 73475 379 73096 1% /dev
tmpfs 75541 266 75275 1% /run
none 75541 3 75538 1% /run/lock
none 75541 1 75540 1% /run/shm

$ find /usr/src/ -type f | wc -l

$ uptime
 06:18:41 up 664 days, 11:16, 2 users, load average: 0.21, 0.20, 0.13

Revision history for this message
algal (alexis) wrote :

(Correction: it's a 12.04 LTS installation, as the console output shows.)

Revision history for this message
pabouk (pabouk) wrote :

@algal If your system is able to remove packages (apt-get remove) do the following:

1. List installed kernel headers packages:
dpkg --get-selections | grep -E '^linux-headers-[1-9].+[[:space:]]install$'

2. Remove the oldest ones (replace ...):
sudo apt-get purge linux-headers-...
sudo apt-get remove linux-headers-...

If the system is in a more serious state (too little free inodes or inconsistent apt database), then move some oldest headers away manually, repair the database, remove some packages as in the point 2 above etc... I described my procedure in the comment here:

Revision history for this message
algal (alexis) wrote :

@pabouk Hi. Thanks so much for the help!

In fact I wasn't able to use apt-get remove, because it tried to do apt-get update first, which failed because of the inode exhaustion.

I wasn't comfortable removing anything manually because I didn't know which files were safe to remove. All but one of the surplus packages in /usr/src were for a linux headers higher than my current linux version, so for all I know the system might depend on them in some way when it tries to upgrade itself from the current version. I erased the one, older package, but this did not free enough inodes to allow any operations to complete.

All of the linux headers were piling up in /usr/src. In the end I resolved the process by doing the following:

1. attaching a much larger new volume, formating it, and mounting it on /mnt
2. cp -a /usr/src/* /mnt # to copy all the linux headers onto the other volume
3. rm -rf /usr/src/* # to remove them from the root volume
3. unmount the new volume mounted at /mnt
4. remount the volume at /usr/src

By this sequence of operations, I exactly preserved the structure of files and folders on the filesystem, while changing how the inode usage was distributed over partitions. So that relieved the inode exhaustion.

Then I was able to use apt-get update, and apt-get autoremove, etc.. to completed the pending upgrades and remove the unused packages.

My big takeaway from this is that I was naive to think unattended-upgrades could run for years unattended, like a router.

Revision history for this message
JohnWashington (ubuntu-johnwash) wrote :

@algal: "My big takeaway from this is that I was naive to think unattended-upgrades could run for years unattended, like a router".

Some might say that the naive ones were the devs who implemented unattended upgrades without thinking through the possible failure scenarios. Of course I couldn't possibly comment!

Don't beat yourself up, Algal! :)

Revision history for this message
Jarno Suni (jarnos) wrote :

Does purging some header package by dpkg --purge work, if the system has ran out of free inodes? As for broken dependencies, it works better than apt-get (provided that you list all depending packages when purging).

Revision history for this message
Jarno Suni (jarnos) wrote :

As for Unattended Upgrades, you can configure it to remove unneeded packages automatically before a dependency problem or an inode problem occur:
I am not sure, if it works on 12.04, though, but I have used it successfully in 14.04 and 16.04 (using the old method).
As for purge-old-kernels script, it has some bugs. It does not purge common header packages (matching ^linux-headers-[0-9.]+-[^-]+$) for instance. I have been developing a more advanced script that can do some troubleshooting, as well. The following video showcases normal purging it can do; the --fix option is not covered by the video:

Revision history for this message
Jarno Suni (jarnos) wrote :

As for #21, I found out that even dpkg --purge needs some free inodes under /var. My script can handle it, if /var is in the same partition as /usr/src; it just removes the selected header version's directories from /usr/src and purges the related versioned kernel packages by dpkg thereafter. I do not see a problem in such a practice.

Revision history for this message
Pablo Piaggio (papibe) wrote :

> ** This bug is no longer a duplicate of bug 1357093

Thank you for this.

Revision history for this message
Jarno Suni (jarnos) wrote :

I think expected behavior would be that Software Updater refuses to start update, and displays error dialog instead, if the update would consume all left inodes. There is a check for free space i.e. self.cache.checkFreeSpace() python method, but apparently it does not catch this case. See also Bug #1460396.

Revision history for this message
Jarno Suni (jarnos) wrote :

I created a high level tool called linux-purge to recover system when this problem appears.

In this case you would run
sudo linux-purge --fix
(It may let you choose which kernel to purge interactively.)

The script can be used for automatic removing of old kernels, if run as cron job or as systemd script (even with older Ubuntu releases). It can be used for checklist based interactive kernel removing, too. It took me a lot of time and effort to write and test that versatile script to save your time and effort. Therefore I have not published the script yet, but I am looking for to get something in return by crowd funding.

Please see here for more information:

If you have a better idea on how to share the expenses, please tell me.

Revision history for this message
WhyteHorse (whytehorse) wrote :

Just crashed my email server... sigh

Revision history for this message
Ben Cates (ben-cates) wrote :

This bug was the root cause of a site outage at my company over the weekend. We're on 16.04.1.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Ben, it would probably be best to file a new bug; this one is filed against update-manager, which I suspect isn't in use on the server in question. Maybe file it against apt, since the /etc/kernel/postinst.d/apt-auto-removal file that should have managed this is owned by apt.


tags: added: rls-bb-incoming
Revision history for this message
Nils Toedtmann (m-launchpad-net-mail-nils-toedtmann-net) wrote :

This report was marked being a duplicate of But it is not specifically about unattended-upgrades; various upgrade mechanisms run into this same issue. Hence this bug report not a duplicate.

Revision history for this message
Brian Murray (brian-murray) wrote :

Bug 1624644 has bug tasks for multiple packages all of which are upgrade mechanisms. This bug was marked as a duplicate of the other, after a meeting of the Ubuntu Foundations team (developers of Ubuntu), because solving that bug report will also resolve this issue. Please let it remain a duplicate.

tags: removed: rls-bb-incoming
Revision history for this message
Pablo Piaggio (papibe) wrote :

@Nils Toedtmann +1. Indeed, this is not about unattended-upgrades.

@Brian Murray, Thanks, I hope you are right, and marking this a duplicate leads to solving this 5+ year bug.


Revision history for this message
Qianqian Fang (fangq) wrote :

happened to me too. plenty of space left on /usr, but no inode left, ended up manually deleting some folders in /usr/src. I thought those are removed when I purge kernels using purge-old-kernels command.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.