Comment 5 for bug 1208393

We just saw this on one of our services as well. No juju errors reported, all units seemed fine but the service was down. Upon closer inspection in the failing unit there seemed to be missing disk space (from the error on the terminal: bash: cannot create temp file for here-document: No space left on device).

du however reported lots of space left, and free also reported enough ram available. Checking for inodes reported 100% usage.

In order to fix this (the unit had more than a handful old kernels still lying around) we first mounted a tmpfs fs on /tmp to work around the issue of apt-get needing to write temporary files.

It looks liked the inode count got full while running some apt operation because apt-get autoremove wouldn't run even after solving the /tmp issue. A quick dpkg --configure -a and apt-get -f install allowed us to apt-get autoremove and get the inode count back down to a more acceptable 19%.