release upgrade failed due to insufficient inodes

[fermulator]

Errors were encountered while processing: /var/cache/apt/archives/linux-headers-3.2.0-58_3.2.0-58.88_all.deb Error in function: A fatal error occurred Please report this as a bug and include the files /var/log/dist-upgrade/main.log and /var/log/dist-upgrade/apt.log in your report. The upgrade has aborted. Your original sources.list was saved in /etc/apt/sources.list.distUpgrade. SystemError: E:Sub-process /usr/bin/dpkg returned an error code (1) No pending crash reports. Try --help for more information. Could not install the upgrades The upgrade has aborted. Your system could be in an unusable state. A recovery will run now (dpkg --configure -a).

[fermulator (fermulator)]

fermulator@fermmy-server:/etc/apt/sources.list.d$ uname -a
Linux fermmy-server 2.6.32-55-generic-pae #117-Ubuntu SMP Tue Dec 3 17:50:05 UTC 2013 i686 i686 i386 GNU/Linux

[fermulator (fermulator)]

(what else do I need to submit here?)

[fermulator (fermulator)]

(also, this is the /2nd/ time I've tried to upgrade, everything was clean before, the first time it borked hard but I didn't record the logs :( -- it left the system in a bad state with apache2 uninstalled, samba configuration wiped ... it was sad :()

[fermulator (fermulator)]

More garbage:
dpkg: error processing python (--configure):
 subprocess installed post-installation script returned error exit status 101
dpkg: dependency problems prevent configuration of python-apport:
 python-apport depends on python (>= 2.7.1-0ubuntu2); however:
  Package python is not configured yet.
 python-apport depends on python (<< 2.8); however:
  Package python is not configured yet.
dpkg: error processing python-apport (--configure):
 dependency problems - leaving unconfigured
Setting up gettext (0.18.1.1-5ubuntu3) ...
dpkg: unrecoverable fatal error, aborting:
 unable to create `/var/lib/dpkg/updates/tmp.i': No space left on device
Traceback (most recent call last):
  File "./xorg_fix_proprietary.py", line 122, in <module>
    filemode='w')
  File "/usr/lib/python2.7/logging/__init__.py", line 1519, in basicConfig
    hdlr = FileHandler(filename, mode)
  File "/usr/lib/python2.7/logging/__init__.py", line 897, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib/python2.7/logging/__init__.py", line 916, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 28] No space left on device: '/var/log/dist-upgrade/xorg_fixup.log'

Upgrade complete

The upgrade has completed but there were errors during the upgrade
process.

[fermulator (fermulator)]

It's freaking out about no space left ... lies!

{{{
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.
fermulator@fermmy-server:/etc/apt/sources.list.d$ sudo dpkg --configure -a
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-3.2.0-58-generic-pae
cp: cannot create regular file `/tmp/mkinitramfs_zLhwgn//lib/modules/3.2.0-58-generic-pae/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko': No space left on device
mkdir: cannot create directory `/tmp/mkinitramfs_zLhwgn//lib/modules/3.2.0-58-generic-pae/kernel/drivers/net/ethernet/chelsio/cxgb': No space left on device
cp: cannot create regular file `/tmp/mkinitramfs_zLhwgn//lib/modules/3.2.0-58-generic-pae/kernel/drivers/net/ethernet/chelsio/cxgb': No space left on device
mkdir: cannot create directory `/tmp/mkinitramfs_zLhwgn//lib/modules/3.2.0-58-generic-pae/kernel/drivers/net/ethernet/stmicro': No space left on device
cp: cannot create regular file `/tmp/mkinitramfs_zLhwgn//lib/modules/3.2.0-58-generic-pae/kernel/drivers/net/ethernet/stmicro/stmmac': No such file or directory
mkdir: cannot create directory `/tmp/mkinitramfs_zLhwgn//lib/modules/3.2.0-58-generic-pae/kernel/drivers/dca': No space left on device
cp: cannot create regular file `/tmp/mkinitramfs_zLhwgn//lib/modules/3.2.0-58-generic-pae/kernel/drivers/dca': No space left on device
}}}

fermulator@fermmy-server:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md1001 9.2G 7.8G 944M 90% /
udev 2.0G 12K 2.0G 1% /dev
tmpfs 785M 532K 784M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
/dev/md1000 2.8T 1.9T 750G 72% /media/arrays/md1000
/dev/md2000 11T 8.7T 1.6T 85% /media/arrays/md2000
/dev/md250 220G 102G 107G 49% /media/arrays/md250
/dev/md640 587G 229G 329G 41% /media/arrays/md640

[fermulator (fermulator)]

I believe I"m able to resolve the "space issue" via
http://askubuntu.com/questions/386282/update-manager-gives-an-error-message
{{{
sudo mv /var/lib/dpkg/status /var/lib/dpkg/status-bad
sudo cp /var/lib/dpkg/status-old /var/lib/dpkg/status
sudo mv /var/lib/dpkg/available /var/lib/dpkg/available-bad
sudo cp /var/lib/dpkg/available-old /var/lib/dpkg/available
sudo rm -rf /var/lib/dpkg/updates/*
sudo rm -rf /var/lib/apt/lists/*
sudo mkdir /var/lib/apt/lists/partial
sudo rm /var/cache/apt/*.bin
sudo apt-get clean
sudo apt-get autoremove
sudo apt-get update
sudo dpkg --configure -a
sudo apt-get install -f
}}}

[fermulator (fermulator)]

Yikes, I didn't know about "-i" on df :
{{{
fermulator@fermmy-server:~$ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/md1001 610800 609388 1412 100% /
udev 206230 763 205467 1% /dev
tmpfs 209054 752 208302 1% /run
none 209054 5 209049 1% /run/lock
none 209054 1 209053 1% /run/shm
/dev/md1000 183148544 1589168 181559376 1% /media/arrays/md1000
/dev/md2000 732569600 19410 732550190 1% /media/arrays/md2000
/dev/md250 14589952 108979 14480973 1% /media/arrays/md250
/dev/md640 39075840 14828 39061012 1% /media/arrays/md640
fermulator@fermmy-server:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/md1001 9.2G 7.1G 1.7G 81% /
udev 2.0G 12K 2.0G 1% /dev
tmpfs 785M 540K 784M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
/dev/md1000 2.8T 1.9T 750G 72% /media/arrays/md1000
/dev/md2000 11T 8.7T 1.6T 85% /media/arrays/md2000
/dev/md250 220G 102G 107G 49% /media/arrays/md250
/dev/md640 587G 229G 329G 41% /media/arrays/md640
}}}

[fermulator (fermulator)]

OK so we've discovered that I ran out of inodes in the middle of the upgrade (AFAICT).

I saw that I had a butt load of linux-headers (stale/old) in /usr/src.
{{{
root@fermmy-server:/usr/src# ls -alt
total 168
drwxr-xr-x 24 root root 4096 Jan 22 21:03 linux-headers-3.2.0-58
drwxrwsr-x 42 root src 4096 Jan 22 21:02 .
drwxr-xr-x 7 root root 4096 Jan 22 20:13 linux-headers-3.2.0-58-generic-pae
drwxr-xr-x 10 root root 4096 Jan 22 20:10 ..
drwxr-xr-x 7 root root 4096 Jan 22 13:03 linux-headers-2.6.32-55-generic-pae
drwxr-xr-x 24 root root 4096 Jan 3 06:55 linux-headers-2.6.32-55
drwxr-xr-x 7 root root 4096 Dec 3 06:47 linux-headers-2.6.32-54-generic-pae
drwxr-xr-x 24 root root 4096 Dec 3 06:47 linux-headers-2.6.32-54
drwxr-xr-x 7 root root 4096 Nov 8 06:30 linux-headers-2.6.32-53-generic-pae
drwxr-xr-x 24 root root 4096 Nov 8 06:30 linux-headers-2.6.32-53
drwxr-xr-x 7 root root 4096 Sep 30 06:48 linux-headers-2.6.32-52-generic-pae
drwxr-xr-x 24 root root 4096 Sep 30 06:48 linux-headers-2.6.32-52
drwxr-xr-x 7 root root 4096 Sep 6 06:46 linux-headers-2.6.32-51-generic-pae
drwxr-xr-x 24 root root 4096 Sep 6 06:45 linux-headers-2.6.32-51
lrwxrwxrwx 1 root root 32 Aug 31 21:07 vboxhost-4.2.16 -> ../share/virtualbox/src/vboxhost
drwxr-xr-x 7 root root 4096 Jul 30 06:44 linux-headers-2.6.32-50-generic-pae
drwxr-xr-x 24 root root 4096 Jul 30 06:44 linux-headers-2.6.32-50
drwxr-xr-x 7 root root 4096 Jul 4 2013 linux-headers-2.6.32-49-generic-pae
drwxr-xr-x 24 root root 4096 Jul 4 2013 linux-headers-2.6.32-49
drwxr-xr-x 7 root root 4096 Jun 14 2013 linux-headers-2.6.32-48-generic-pae
drwxr-xr-x 24 root root 4096 Jun 14 2013 linux-headers-2.6.32-48
drwxr-xr-x 7 root root 4096 May 15 2013 linux-headers-2.6.32-47-generic-pae
drwxr-xr-x 24 root root 4096 May 15 2013 linux-headers-2.6.32-47
drwxr-xr-x 7 root root 4096 Apr 19 2013 linux-headers-2.6.32-46-generic-pae
drwxr-xr-x 24 root root 4096 Apr 19 2013 linux-headers-2.6.32-46
drwxr-xr-x 7 root root 4096 Feb 22 2013 linux-headers-2.6.32-45-generic-pae
drwxr-xr-x 24 root root 4096 Feb 22 2013 linux-headers-2.6.32-45
drwxr-xr-x 7 root root 4096 Oct 11 2012 linux-headers-2.6.32-44-generic-pae
drwxr-xr-x 24 root root 4096 Oct 11 2012 linux-headers-2.6.32-44
drwxr-xr-x 7 root root 4096 Oct 2 2012 linux-headers-2.6.32-41-generic-pae
drwxr-xr-x 24 root root 4096 Oct 2 2012 linux-headers-2.6.32-41
drwxr-xr-x 7 root root 4096 Sep 18 2012 linux-headers-2.6.32-43-generic-pae
drwxr-xr-x 24 root root 4096 Sep 18 2012 linux-headers-2.6.32-43
drwxr-xr-x 7 root root 4096 Sep 5 2012 linux-headers-2.6.32-42-generic-pae
drwxr-xr-x 24 root root 4096 Sep 5 2012 linux-headers-2.6.32-42
drwxr-xr-x 7 root root 4096 Mar 23 2012 linux-headers-2.6.32-40-generic-pae
drwxr-xr-x 24 root root 4096 Mar 23 2012 linux-headers-2.6.32-40
drwxr-xr-x 7 root root 4096 Mar 6 2012 linux-headers-2.6.32-39-generic-pae
drwxr-xr-x 24 root root 4096 Mar 6 2012 linux-headers-2.6.32-39
drwxr-xr-x 7 root root 4096 Jan 24 2012 linux-headers-2.6.32-38-generic-pae
drwxr-xr-x 24 root root 4096 Jan 24 2012 linux-headers-2.6.32-38
drwxr-xr-x 7 root root 4096 Dec 20 2011 linux-headers-2.6.32-37-generic...

[fermulator (fermulator)]

To help users in the future, I've updated the wiki: https://help.ubuntu.com/community/PreciseUpgrades#Before_You_Start to include "make sure you have enough disk space" (although I wish I knew the actual requirements here)....

[fermulator (fermulator)]

This is crazy ; how is the system "done"? (the upgrade seemed to fail terribly?)

fermulator@fermmy-server:~$ sudo do-release-upgrade --mode=server
Checking for a new Ubuntu release
No new release found
fermulator@fermmy-server:~$ sudo apt-get dist-upgrade -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

[fermulator (fermulator)]

Logs indicate it failed around here

[Errno 28] No space left on device: '/usr/lib/python2.7/dist-packages/numpy/core/tests/test_print.pyc'
dpkg: error processing python (--configure):
 subprocess installed post-installation script returned error exit status 101
dpkg: dependency problems prevent configuration of python-apport:
 python-apport depends on python (>= 2.7.1-0ubuntu2); however:
  Package python is not configured yet.
 python-apport depends on python (<< 2.8); however:
  Package python is not configured yet.
dpkg: error processing python-apport (--configure):
 dependency problems - leaving unconfigured Setting up gettext (0.18.1.1-5ubuntu3) ... dpkg: unrecoverable fatal error, aborting: unable to create `/var/lib/dpkg/updates/tmp.i': No space left on device Traceback (most recent call last): File "./xorg_fix_proprietary.py", line 122, in <module> filemode='w') File "/usr/lib/python2.7/logging/__init__.py", line 1519, in basicConfig hdlr = FileHandler(filename, mode) File "/usr/lib/python2.7/logging/__init__.py", line 897, in __init__ StreamHandler.__init__(self, self._open()) File "/usr/lib/python2.7/logging/__init__.py", line 916, in _open stream = open(self.baseFilename, self.mode) ...

[fermulator (fermulator)]

I think the suggestion here would be for the tool to prevent an upgrade if there isn't enough disk space (or in my case, inodes... ;o)