root file system 100% full, goes 52% after reboot

[Sergei Vorobyov]

I am running several I/O intensive applications creating and deleting many files on the separate /home partition. After about a day of operation my root file system / containing everything except /home goes 100% full, although running ncdu -x from / shows just about 50% fullness. 100% goes down to (correct) 52% after a reboot. Note that all my files are created within the mounted /home partition. I do not run any specific process as a root (beyond those of a stock Ubuntu 14.04 distribution).

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 23G 12G 11G 52% /
/dev/sda6 98G 67G 27G 72% /home

1)
$ lsb_release -rd
Description: Ubuntu 14.04 LTS
Release: 14.04

2)
$ apt-cache policy e2fslibs
e2fslibs:
  Installed: 1.42.9-3ubuntu1
  Candidate: 1.42.9-3ubuntu1
  Version table:
 *** 1.42.9-3ubuntu1 0
        500 http://se.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status

3) I expect my file systems to be cleaned up withoutr rebooting

4) the root file system overflows just after 1 day of normal operation

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: e2fslibs 1.42.9-3ubuntu1
ProcVersionSignature: Ubuntu 3.13.0-24.47-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu3
Architecture: amd64
CurrentDesktop: Unity
Date: Thu May 15 09:04:23 2014
Dependencies:
 gcc-4.9-base 4.9-20140406-0ubuntu1
 libc6 2.19-0ubuntu6
 libgcc1 1:4.9-20140406-0ubuntu1
 multiarch-support 2.19-0ubuntu6
InstallationDate: Installed on 2014-02-10 (93 days ago)
InstallationMedia: Ubuntu 13.10 "Saucy Salamander" - Release amd64 (20131016.1)
SourcePackage: e2fsprogs
UpgradeStatus: Upgraded to trusty on 2014-04-22 (22 days ago)

[Sergei Vorobyov (svorobyov)]

now 5 hours from my initial report

#df -h

hangs *indefinetely*,

whereas ncdu -x from / shows about 1/2 fullness (my root patition / is about 23G)

    9,3GiB [##########] /usr
    1,3GiB [# ] /var
  451,7MiB [ ] /lib
   62,8MiB [ ] /boot
   15,8MiB [ ] /etc
   12,2MiB [ ] /sbin
    9,6MiB [ ] /bin
    5,3MiB [ ] /root
  152,0KiB [ ] /tmp
   20,0KiB [ ] /mnt
e 16,0KiB [ ] /lost+found
    8,0KiB [ ] /media
    4,0KiB [ ] /lib64
e 4,0KiB [ ] /srv
e 4,0KiB [ ] /opt
    4,0KiB [ ] /export
e 4,0KiB [ ] /cdrom
@ 0,0 B [ ] initrd.img.old
@ 0,0 B [ ] initrd.img
@ 0,0 B [ ] vmlinuz.old
@ 0,0 B [ ] vmlinuz
> 0,0 B [ ] /sys
> 0,0 B [ ] /run
> 0,0 B [ ] /proc
> 0,0 B [ ] /home
> 0,0 B [ ] /dev

# fdisk -l /dev/sda

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x3de5d7d1

   Device Boot Start End Blocks Id System
/dev/sda1 * 2048 206847 102400 7 HPFS/NTFS/exFAT
/dev/sda2 206848 710531071 355162112 7 HPFS/NTFS/exFAT
/dev/sda3 710531072 759359196 24414062+ 83 Linux
/dev/sda4 759361534 976771071 108704769 5 Extended
/dev/sda5 759361536 768149503 4393984 82 Linux swap / Solaris
/dev/sda6 768151552 976771071 104309760 83 Linux

[Theodore Ts'o (tytso)]

If you have processes keeping a file open, the space doesn't get reclaimed until the file is closed or the process is killed. Obviously, the latter tends to happen when you reboot a system. This is standard practice going back to 1970's and Unix; in fact, it is required by the POSIX standard.

Why df is hanging is a separate issue. Creating a separate bug against the kernel and including the output of "strace df -h" might be helpful.

[Sergei Vorobyov (svorobyov)]

1. I don't keep many files open forever, they are duly closed after being used. Many created, filled up, closed, uploaded, and erased.

2. FIles get created on /home

3. / get overflown, not /home

4. If your explanation were correct, /home would have been overflown.

5. I think you explanation (not my report) is invalid. Please return the status to "to be investigated/explained"

[Sergei Vorobyov (svorobyov)]

strace dies as well

[Egmont Koblinger (egmont-gmail)]

> 1. I don't keep many files open forever, they are duly closed after being used. Many created, filled up, closed, uploaded, and erased.
> 2. FIles get created on /home
> 3. / get overflown, not /home
> 4. If your explanation were correct, /home would have been overflown.

Sergei, you yourself have found the answer to this (bug 1320157), indeed there was someone writing to / which made it full.

5. I think you explanation (not my report) is invalid. Please return the status to "to be investigated/explained"

The df hang needs investigation (in a separate bugreport). The rest is working as intended, as Theodore said.