Empty tmpfs mount reported as full

[Ken Sharp]

After using a tmpfs mount for a while it starts reporting that it is almost full and become unusable.

$ mount ; df ; sudo ls -alR /tmp/test/

<snip>
none on /tmp/test type tmpfs (rw,uid=1000,gid=1000)
none on /home/ken/.wine type aufs (rw,br:/tmp/test:/home/ken/.wine.dotnet20)

Filesystem Size Used Avail Use% Mounted on
<snip>
none 1003M 881M 122M 88% /tmp/test
none 1003M 881M 122M 88% /home/ken/.wine

/tmp/test/:
total 12
drwxrwxrwt 4 ken ken 100 Aug 6 20:35 .
drwxrwxrwt 10 root root 12288 Aug 6 20:38 ..
-r--r--r-- 1 root root 0 Aug 6 19:08 .wh..wh.aufs
drwx------ 2 root root 40 Aug 6 19:08 .wh..wh.orph
drwx------ 2 root root 40 Aug 6 19:08 .wh..wh.plnk

/tmp/test/.wh..wh.orph:
total 0
drwx------ 2 root root 40 Aug 6 19:08 .
drwxrwxrwt 4 ken ken 100 Aug 6 20:35 ..

/tmp/test/.wh..wh.plnk:
total 0
drwx------ 2 root root 40 Aug 6 19:08 .
drwxrwxrwt 4 ken ken 100 Aug 6 20:35 ..

As you should be able to see the tmpfs mount on /tmp/test claims to be using ~900 MB but it's really only using a few bytes. It works perfectly for a while but then does this.

I have been using the mount with aufs and the copy-on-write files are stored here. Since starting this report the "used" amount has dropped around 20 MB but there's no reason for this. There is no longer anything running that would use this directory.

Nothing seems relevant in dmesg or the syslog.

I don't know how to debug further.

I will check the upstream kernel, but this is likely to cause other problems and will probably be difficult to recreate. I will report back once I know.

Note to self: While testing http://bugs.winehq.org/show_bug.cgi?id=34217

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: linux-image-3.2.0-51-generic-pae 3.2.0-51.77
ProcVersionSignature: Ubuntu 3.2.0-51.77-generic-pae 3.2.48
Uname: Linux 3.2.0-51-generic-pae i686
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.24.
AplayDevices: aplay: device_list:252: no soundcards found...
ApportVersion: 2.0.1-0ubuntu17.3
Architecture: i386
ArecordDevices: arecord: device_list:252: no soundcards found...
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/by-path', '/dev/snd/controlC0', '/dev/snd/hwC0D0', '/dev/snd/hwC0D1', '/dev/snd/pcmC0D0c', '/dev/snd/pcmC0D0p', '/dev/snd/pcmC0D1p', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
Date: Tue Aug 6 20:38:16 2013
HibernationDevice: RESUME=UUID=113e73a6-3c10-4fbc-a9a4-5160e676d325
InstallationMedia: Ubuntu-Server 12.04.1 LTS "Precise Pangolin" - Release i386 (20120817.3)
Lsusb:
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: Dell Inc. MM061
MarkForUpload: True
ProcEnviron:
 LANGUAGE=en_GB:en
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
ProcFB: 0 inteldrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.2.0-51-generic-pae root=UUID=0c3228e5-3952-43c1-b666-1c790f791025 ro persistent zcache
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-3.2.0-51-generic-pae N/A
 linux-backports-modules-3.2.0-51-generic-pae N/A
 linux-firmware 1.79.6
RfKill: Error: [Errno 2] No such file or directory
SourcePackage: linux
StagingDrivers: zram
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 06/13/2007
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A17
dmi.board.name: 0KD882
dmi.board.vendor: Dell Inc.
dmi.chassis.type: 8
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA17:bd06/13/2007:svnDellInc.:pnMM061:pvr:rvnDellInc.:rn0KD882:rvr:cvnDellInc.:ct8:cvr:
dmi.product.name: MM061
dmi.sys.vendor: Dell Inc.

[Ken Sharp (kennybobs)]

Hmmm...

I just released the aufs mount and the tmpfs is correctly showing as empty again.

$ sudo umount ~/.wine
$ df
Filesystem Size Used Avail Use% Mounted on
<snip>
none 1003M 0 1003M 0% /tmp/test

Maybe aufs is messing something up?

[Brad Figg (brad-figg)]

This change was made by a bot.

[Ken Sharp (kennybobs)]

Okay, I think I can recreate this fairly reliably.

mkdir /tmp/original
mkdir /tmp/cow
mkdir /tmp/mountpoint

mount -t tmpfs none /tmp/mountpoint
mount -t aufs -o br:/tmp/cow:/tmp/original none /tmp/mountpoint

for j in {1..15} ; do dd count=$RANDOM if=/dev/zero of=/tmp/original/data$j bs=5120 ; done

for i in {1..150} ; do echo $i ; rm /tmp/mountpoint/* 2>&- ; (for j in {1..15} ; do dd count=$RANDOM if=/dev/zero of=/tmp/mountpoint/data$j bs=5120 ; done) ; done

rm /tmp/cow

The for j statement is to create some original files that will be changed. I just had it create random files.

The for i statement does the same thing but so that the new files are in the copy-on-write folder. I tried creating one large file but it was hard to recreate, so I used the loop to create a bunch of random-size files, then remove them, then do it again.

Once cleaned I'm still left with an apparently full mount despite the fact I have removed all the new files:

none 1003M 955M 48M 96% /tmp/cow
none 1003M 955M 48M 96% /tmp/mountpoint

There's a number of different issues that pop up here:

1. rm /tmp/mountpoint/* should be removing the new files I have created. They are not being removed. rm reports that these files cannot be removed. They should be removed and either show an empty directory, or show the original files. Either way there should be some logic here that results in those files being removed.

2. rm /tmp/cow removes the files but aufs/tmpfs do not see the change, even though ls /tmp/cow and ls /tmp/mountpoint show empty directories.

There's something very wrong here.

My original test is slightly different in that it removed /tmp/cow/* in the loop. For example:

for i in {1..150} ; do echo $i ; rm /tmp/cow/* 2>&- ; (for j in {1..15} ; do dd count=$RANDOM if=/dev/zero of=/tmp/mountpoint/data$j bs=5120 ; done) ; done

Either way aufs/tmpfs seem to pick up the change for a while, then give up and simply report the mount full.

Will try the mainline kernels tomorrow.

[Ken Sharp (kennybobs)]

Both the mainline kernel and the daily build don't support aufs. There's nothing that mentions aufs in the build log at all. I assume aufs support has not been built.

Mainline:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.2.48-precise/

Daily:
http://kernel.ubuntu.com/~kernel-ppa/mainline/daily/current/
linux-headers-3.11.0-999-generic_3.11.0-999.201308070514_i386.deb 07-Aug-2013 09:32 1.0M
linux-headers-3.11.0-999_3.11.0-999.201308070514_all.deb 07-Aug-2013 09:16 12M
linux-image-3.11.0-999-generic_3.11.0-999.201308070514_i386.deb 07-Aug-2013 09:32 46M

[Joseph Salisbury (jsalisbury)]

Did this issue start happening after an update/upgrade? Was there a kernel version where you were not having this particular problem? This will help determine if the problem you are seeing is the result of the introduction of a regression, and when this regression was introduced. If this is a regression, we can perform a kernel bisect to identify the commit that introduced the problem.

[Ken Sharp (kennybobs)]

I'm sorry but I honestly don't know.

I saw this problem a few times around a year ago on a Precise server but I never got chance to look into it. So it's been around for at least a year.