du/df/lsof disagree about free disk space

[dialtone]

The past 2 releases of Ubuntu on Ec2 have been filling up their root filesystem over a period of about 3-4 weeks. However a simple reboot of the instance brings the disk back to 14% use, because I imagine it runs fsck. The fs errors happen without any unexpected reboot or restart of sorts.

'sudo lsof /dev/xvda1 | grep -i deleted' is empty.

df -h
deploy@--:/$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 9.9G 5.0G 4.5G 53% /
none 3.7G 116K 3.7G 1% /dev
none 3.7G 0 3.7G 0% /dev/shm
none 3.7G 52K 3.7G 1% /var/run
none 3.7G 0 3.7G 0% /var/lock
/dev/xvdb 414G 500M 392G 1% /mnt

sudo du -h --max-depth=1
19M ./root
53M ./srv
257M ./var
92K ./home
4.0K ./selinux
748M ./usr
7.7M ./bin
4.0K ./opt
0 ./sys
6.1M ./etc
9.5M ./sbin
[lots of]
du: cannot access `./proc/2008/task/1234/fd/0987': No such file or directory
0 ./proc
116K ./dev
72M ./lib
285M ./mnt
273M ./tmp
4.0K ./nonexistent
16K ./lost+found
15M ./boot
1.7G .

The version of du has also been compiled from scratch and reports the same data.

sudo fsck.ext3 -n /dev/xvda1 reports:
e2fsck 1.41.14 (22-Dec-2010)
Warning! /dev/xvda1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
uec-rootfs contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found. Fix? no

Inode 24577 was part of the orphaned inode list. IGNORED.
Inode 24578 was part of the orphaned inode list. IGNORED.
Inode 24579 was part of the orphaned inode list. IGNORED.
Inode 24580 was part of the orphaned inode list. IGNORED.
Inode 24581 was part of the orphaned inode list. IGNORED.
Inode 24582 was part of the orphaned inode list. IGNORED.
Inode 24583 was part of the orphaned inode list. IGNORED.
Inode 82238 was part of the orphaned inode list. IGNORED.
Deleted inode 82239 has zero dtime. Fix? no

Inode 82240 was part of the orphaned inode list. IGNORED.
Inode 82241 was part of the orphaned inode list. IGNORED.
Inode 82242 was part of the orphaned inode list. IGNORED.
Inode 82243 was part of the orphaned inode list. IGNORED.
Inode 82244 was part of the orphaned inode list. IGNORED.
Inode 82245 was part of the orphaned inode list. IGNORED.
Inode 82246 was part of the orphaned inode list. IGNORED.
Inode 82247 was part of the orphaned inode list. IGNORED.
Inode 82248 was part of the orphaned inode list. IGNORED.
Inode 82249 was part of the orphaned inode list. IGNORED.
Inode 82250 was part of the orphaned inode list. IGNORED.
Inode 90119 was part of the orphaned inode list. IGNORED.
Inode 90120 was part of the orphaned inode list. IGNORED.
Inode 90121 was part of the orphaned inode list. IGNORED.
Inode 90122 was part of the orphaned inode list. IGNORED.
Inode 90123 was part of the orphaned inode list. IGNORED.
Inode 90124 was part of the orphaned inode list. IGNORED.
Inode 90125 was part of the orphaned inode list. IGNORED.
Inode 90126 was part of the orphaned inode list. IGNORED.
Inode 90127 was part of the orphaned inode list. IGNORED.
Inode 90128 was part of the orphaned inode list. IGNORED.
Inode 90129 was part of the orphaned inode list. IGNORED.
Inode 90130 was part of the orphaned inode list. IGNORED.
Inode 90131 was part of the orphaned inode list. IGNORED.
Inode 90132 was part of the orphaned inode list. IGNORED.
Inode 90133 was part of the orphaned inode list. IGNORED.
Inode 90134 was part of the orphaned inode list. IGNORED.
Inode 90135 was part of the orphaned inode list. IGNORED.
Inode 90136 was part of the orphaned inode list. IGNORED.
Inode 90137 was part of the orphaned inode list. IGNORED.
Inode 90138 was part of the orphaned inode list. IGNORED.
Inode 90139 was part of the orphaned inode list. IGNORED.
Inode 90140 was part of the orphaned inode list. IGNORED.
Inode 90141 was part of the orphaned inode list. IGNORED.
Inode 90142 was part of the orphaned inode list. IGNORED.
Inode 155649 was part of the orphaned inode list. IGNORED.
Inode 155650 was part of the orphaned inode list. IGNORED.
Inode 155651 was part of the orphaned inode list. IGNORED.
Inode 155652 was part of the orphaned inode list. IGNORED.
Inode 155653 was part of the orphaned inode list. IGNORED.
Inode 155654 was part of the orphaned inode list. IGNORED.
Inode 155655 was part of the orphaned inode list. IGNORED.
Inode 155656 was part of the orphaned inode list. IGNORED.
Inode 155657 was part of the orphaned inode list. IGNORED.
Inode 306018 was part of the orphaned inode list. IGNORED.
Inode 306019 was part of the orphaned inode list. IGNORED.
Inode 306020 was part of the orphaned inode list. IGNORED.
Inode 306021 was part of the orphaned inode list. IGNORED.
Inode 306022 was part of the orphaned inode list. IGNORED.
Inode 358495 was part of the orphaned inode list. IGNORED.
Inode 358496 was part of the orphaned inode list. IGNORED.
Inode 358497 was part of the orphaned inode list. IGNORED.
Inode 358498 was part of the orphaned inode list. IGNORED.
Inode 358499 was part of the orphaned inode list. IGNORED.
Inode 358500 was part of the orphaned inode list. IGNORED.
Inode 358501 was part of the orphaned inode list. IGNORED.
Inode 358502 was part of the orphaned inode list. IGNORED.
Inode 358503 was part of the orphaned inode list. IGNORED.
Inode 358504 was part of the orphaned inode list. IGNORED.
Inode 358505 was part of the orphaned inode list. IGNORED.
Inode 417793 was part of the orphaned inode list. IGNORED.
Inode 417794 was part of the orphaned inode list. IGNORED.
Inode 417795 was part of the orphaned inode list. IGNORED.
Inode 417796 was part of the orphaned inode list. IGNORED.
Inode 417797 was part of the orphaned inode list. IGNORED.
Inode 417798 was part of the orphaned inode list. IGNORED.
Inode 417799 was part of the orphaned inode list. IGNORED.
Inode 417800 was part of the orphaned inode list. IGNORED.
Inode 417801 was part of the orphaned inode list. IGNORED.
Inode 417802 was part of the orphaned inode list. IGNORED.
Inode 417803 was part of the orphaned inode list. IGNORED.
Inode 417804 was part of the orphaned inode list. IGNORED.
Inode 417805 was part of the orphaned inode list. IGNORED.
Inode 417806 was part of the orphaned inode list. IGNORED.
Inode 417807 was part of the orphaned inode list. IGNORED.
Inode 417808 was part of the orphaned inode list. IGNORED.
Inode 417809 was part of the orphaned inode list. IGNORED.
Inode 417810 was part of the orphaned inode list. IGNORED.
Inode 417811 was part of the orphaned inode list. IGNORED.
Inode 417812 was part of the orphaned inode list. IGNORED.
Inode 417813 was part of the orphaned inode list. IGNORED.
Inode 417814 was part of the orphaned inode list. IGNORED.
Inode 417815 was part of the orphaned inode list. IGNORED.
Inode 417816 was part of the orphaned inode list. IGNORED.
Inode 417817 was part of the orphaned inode list. IGNORED.
Inode 417818 was part of the orphaned inode list. IGNORED.
Inode 417819 was part of the orphaned inode list. IGNORED.
Inode 417820 was part of the orphaned inode list. IGNORED.
Inode 417821 was part of the orphaned inode list. IGNORED.
Inode 548865 was part of the orphaned inode list. IGNORED.
Inode 548866 was part of the orphaned inode list. IGNORED.
Inode 548867 was part of the orphaned inode list. IGNORED.
Inode 548868 was part of the orphaned inode list. IGNORED.
Inode 548869 was part of the orphaned inode list. IGNORED.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -(99459--131071) -(131586--157284) -(328217--329727) -(329749--331775) -(331783--333823) -(333834--335871) -(335879--337919) -(337925--339967) -(339983--342015) -(342025--344063) -(344069--346111) -(346125--348159) -(348168--348171) -(348174--350207) -350281 -(355173--356351) -(356370--358399) -(358408--360447) -(360962--370687) -(370689--393215) -(393730--403455) -(403457--425983) -(426498--458751) -(459266--491519) -(492034--524287) -(524802--557055) -(557570--589823) -(590338--622591) -(623106--655359) -(655874--688127) -(688642--720895) -(721416--723305) -(1215919--1222655) -(1222657--1234943) -(1235188--1236991) -(1255257--1267711) -(1267786--1269759) -(1276217--1277951) -(1311781--1318143) -(1412292--1431551) -(1431736--1433599) -(1440995--1441791) -(1442306--1464319) -(1464681--1466367) -(1475530--1497087) -(1497337--1499135) -(1510345--1529855) -(1529904--1531903) -(1533057--1535999) -(1536040--1536048) -(1536055--1536064) -(1536075--1536080) -(1536114--1538036) -(1671682--1703935) -(1704450--1736703) -(1737218--1759231) -(1759233--1761279) -(1763249--1769471) -(1769986--1802239) -(1802754--1835007) -(1835522--1867775) -(1868290--1900543) -(1901058--1923071) -(1923080--1929292) -(2195970--2228223) -(2228738--2231296) -(2232328--2240989)
Fix? no

Free blocks count wrong for group #70 (21, counted=20).
Fix? no

Free blocks count wrong (2285691, counted=1294884).
Fix? no

Inode bitmap differences: -(24577--24583) -(82238--82250) -(90119--90142) -(155649--155657) -(306018--306022) -(358495--358505) -(417793--417821) -(548865--548869)
Fix? no

Free inodes count wrong (605146, counted=604571).
Fix? no

uec-rootfs: ********** WARNING: Filesystem still has errors **********

uec-rootfs: 50214/655360 files (0.2% non-contiguous), 335749/2621440 blocks

[dialtone (dialtone)]

Also:

sudo find /proc -type f -links 0 -ls

returns

find: `/proc/1015/fdinfo/305': No such file or directory
find: `/proc/1016/fd/1798': No such file or directory
find: `/proc/1016/fd/1802': No such file or directory
find: `/proc/1016/fd/1804': No such file or directory
find: `/proc/1016/fdinfo/552': No such file or directory
find: `/proc/1016/fdinfo/763': No such file or directory
find: `/proc/1016/fdinfo/1082': No such file or directory
find: `/proc/1016/fdinfo/1798': No such file or directory
find: `/proc/2008/task/2008/fdinfo/2293': No such file or directory
find: `/proc/2008/task/2009/fd/1239': No such file or directory
find: `/proc/2008/task/2009/fdinfo/2404': No such file or directory
find: `/proc/2008/task/2010/fd/3166': No such file or directory
find: `/proc/2008/task/2010/fd/3172': No such file or directory
find: `/proc/2008/task/2010/fd/3173': No such file or directory
find: `/proc/2008/task/2010/fdinfo/2293': No such file or directory
find: `/proc/2008/task/2010/fdinfo/3157': No such file or directory
find: `/proc/2008/task/2010/fdinfo/3160': No such file or directory
find: `/proc/2008/task/2010/fdinfo/3169': No such file or directory
find: `/proc/2008/task/2011/fd/2757': No such file or directory
find: `/proc/2008/task/2011/fdinfo/1239': No such file or directory
find: `/proc/2008/task/2012/fdinfo/2293': No such file or directory
find: `/proc/2008/task/2012/fdinfo/2404': No such file or directory
find: `/proc/2008/task/2012/fdinfo/3139': No such file or directory
find: `/proc/2008/task/2014/fd/2904': No such file or directory
find: `/proc/2008/task/2014/fdinfo/2293': No such file or directory
find: `/proc/2008/task/2015/fd/2293': No such file or directory
find: `/proc/2008/task/2015/fd/2404': No such file or directory
find: `/proc/2008/task/2015/fd/2757': No such file or directory
find: `/proc/2008/task/2015/fd/3139': No such file or directory
find: `/proc/2008/task/2015/fdinfo/2404': No such file or directory
find: `/proc/2008/task/2018/fdinfo/3139': No such file or directory
find: `/proc/2008/task/2018/fdinfo/3160': No such file or directory
find: `/proc/2008/task/2018/fdinfo/3183': No such file or directory
find: `/proc/2008/task/2018/fdinfo/3190': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3139': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3166': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3169': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3177': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3178': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3181': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3183': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3184': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3190': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3193': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3202': No such file or directory
find: `/proc/2008/task/2020/fdinfo/3246': No such file or directory
find: `/proc/2008/task/2021/fd/2293': No such file or directory
find: `/proc/2008/task/2021/fd/2757': No such file or directory
find: `/proc/2008/task/20...