du reports existing files as missing

[Jimmy P]

Binary package hint: coreutils

running du --max-depth=1 produces the following:

emmy:/hd/sda2$ du --max-depth=1
641120 ./i386
10880 ./valueadd
40768 ./dotnetfx
654080 ./uttad
2976 ./book
224 ./sysinfo
3398720 ./windows
19883360 ./Documents and Settings
4394624 ./Program Files
du: `./System Volume Information': No such file or directory
du: `./found.000': No such file or directory
du: `./Recycled': No such file or directory
du: `./found.001': No such file or directory
du: `./src': No such file or directory
du: `./proj1': No such file or directory
du: `./test': No such file or directory
du: `./Acer': No such file or directory
du: `./found.002': No such file or directory
du: `./found.003': No such file or directory
du: `./explore2fs-1.07': No such file or directory
du: `./pics': No such file or directory
du: `./schoolpictures': No such file or directory
du: `./doj': No such file or directory
du: `./writable': No such file or directory
du: `./taxes': No such file or directory
du: `./risk': No such file or directory
du: `./gettingstarted_files': No such file or directory
du: `./ppt': No such file or directory
du: `./transunion_files': No such file or directory
du: `./apache': No such file or directory
du: `./dwl1000': No such file or directory
du: `./mirc': No such file or directory
du: `./metron': No such file or directory
du: `./install.itc': No such file or directory
du: `./ai': No such file or directory
du: `./httpserver_files': No such file or directory
du: `./mille': No such file or directory
du: `./web': No such file or directory
du: `./lawjobs': No such file or directory
du: `./winrisk': No such file or directory
31074240 .

all those directories exist though, e.g.:

emmy:/hd/sda2$ ls -ld winrisk/ lawjobs/
drwxrwx--- 2 root plugdev 32768 2006-01-14 17:15 lawjobs//
drwxrwx--- 2 root plugdev 32768 2006-01-16 01:58 winrisk//

/hd/sda2 is a windows partition:
emmy:/hd/sda2$ mount |grep sda2
/dev/sda2 on /hd/sda2 type vfat (rw,utf8,umask=007,gid=46)

the same things happens without --max-depth=1, I just used that for readability

[Jimmy P (jimmypoor2004)]

sorry for the spam, this is on ubuntu 6.06

[Jimmy P (jimmypoor2004)]

I get the same behavior on edgy on a different machine.

[Ralph Janke (txwikinger)]

I have found a similar behaviour using du on a cifs from Windows mounted directory, although only with files that contain spaces in their file names.

On ext3 filesystem I had no problems with such files.

Can you give some more information about the filesystem, mounting, as well the kind of file names for which this behaviour occurs.

Thanks,

[Ralph Janke (txwikinger)]

Could you also test if this happens when you mount the drive without UTF8 as an option.

[Ralph Janke (txwikinger)]

I can confirm and reproduce this behaviour on dapper if unregular characters are part of the file name (i.e. a telephone symbol from wingdings font).

[Jimmy P (jimmypoor2004)]

this is on edgy -

mount |grep windows:
/dev/hda1 on /windows type vfat (rw,umask=007,gid=46,utf8)
cindy:/windows$ du --max-depth=1
19456 ./backup
465088 ./i386
50560 ./dell
23296 ./discover
2077312 ./winnt
15136 ./recycled
22112 ./drivers
1376 ./mb
970912 ./Documents and Settings
7163584 ./Program Files
508800 ./FPGAdv53
du: `./car': No such file or directory
du: `./Media': No such file or directory
du: `./ext2fs': No such file or directory
du: `./wpa_files': No such file or directory
du: `./router': No such file or directory
du: `./My Music': No such file or directory
du: `./monitor': No such file or directory
du: `./home': No such file or directory
38944 ./stuff
du: `./wpa2_files': No such file or directory
du: `./WUTemp': No such file or directory
du: `./war2': No such file or directory
du: `./sblive': No such file or directory
41312 ./brian
du: `./wpa3_files': No such file or directory
13984 ./taxes
.
.
.

The first directory it couldn't find was called "car". After that it can't find most directories but can find some.

After remounting without utf8:
mount | grep windows:
/dev/hda1 on /windows type vfat (rw,umask=007,gid=46)

cindy:/windows$ du --max-depth=1
19456 ./backup
465088 ./i386
50560 ./dell
23296 ./discover
2077312 ./winnt
15136 ./recycled
22112 ./drivers
1376 ./mb
970912 ./Documents and Settings
7163584 ./Program Files
508800 ./FPGAdv53
1120 ./car
64 ./Media
960 ./ext2fs
1024 ./wpa_files
736 ./router
32 ./My Music
224 ./monitor
32 ./home
38944 ./stuff
1184 ./wpa2_files
32 ./WUTemp
28416 ./war2
3456 ./sblive
41312 ./brian
96 ./wpa3_files
13984 ./taxes
1184 ./wpa4_files
3392 ./k
960 ./ext2
155232 ./oo
3008 ./My Installations
2080 ./printer
13056 ./serial
27488 ./printerwin
211264 ./perl
9760 ./test
928 ./rightcpu_files
576 ./middle_files
576 ./left_files
248096 ./law
du: `./lawapps': No such file or directory
du: `./Temp': No such file or directory
du: `./printergui': No such file or directory
du: `./ap': No such file or directory
du: `./printerGuiSDI': No such file or directory
du: `./delldrivers': No such file or directory
du: `./24 Season 1': No such file or directory
du: `./testuchar': No such file or directory
du: `./testendian': No such file or directory
14162944 .

[Micah Cowan (micahcowan)]

This sounds to be a vfat driver issue rather than a du issue; I've added the linux kernel package for this bug.

[Ralph Janke (txwikinger)]

Does cifs use vfat? It happens also with cifs mounts

[Micah Cowan (micahcowan)]

Well, but it still sounds like a likely kernel issue, to me, if the deciding factor is whether or not it's mounted in utf-8 mode.

I have seen very similar issues on corrupted filesystems, when ls works but ls -l doesn't (readdir() returns the name, but stat() can't find it). I suspect you'd have the same problem ls -l: if you do, then it would definitely be evidence for a kernel, as opposed to a program, bug.

Could you please verify this for me?

[Micah Cowan (micahcowan)]

Er, obviously I misread how much of a deciding factor utf8 is; however, I'd still be interested in seeing what the success is like with ls -l.

[Jimmy P (jimmypoor2004)]

A couple observations:

1) When du can't find a directory, going into that directory and executing a du or an ls causes subsequent invocations of du to find that directory
2) ls -l seems pretty normal

evidence:
cindy:/windows$ du --max-depth=1
19456 ./backup
.
.
1120 ./car
du: `./Media': No such file or directory
du: `./ext2fs': No such file or directory
.
.

cindy:/windows$ cd Media/
cindy:/windows/Media$ du --max-depth=1
32 ./Music
64 .
cindy:/windows/Media$ cd ..
cindy:/windows$ du --max-depth=1
19456 ./backup
.
.
1120 ./car
64 ./Media
du: `./ext2fs': No such file or directory
cindy:/windows$ ls -ld ext2fs
drwxrwx--- 2 root plugdev 32768 2004-05-29 13:33 ext2fs/
cindy:/windows$ cd ext2fs/
cindy:/windows/ext2fs$ ls -l
total 928
-rwxrwx--- 1 root plugdev 6724 2002-04-20 19:24 changes.txt*
-rwxrwx--- 1 root plugdev 17984 1998-06-19 19:29 copying.txt*
-rwxrwx--- 1 root plugdev 25956 1999-07-22 00:45 diskio2.dll*
-rwxrwx--- 1 root plugdev 527 2006-03-31 14:35 explore2fs debug log.txt*
-rwxrwx--- 1 root plugdev 780800 2002-10-20 16:56 explore2fs.exe*
-rwxrwx--- 1 root plugdev 6263 2000-07-11 23:24 readme.txt*

Now that we've "looked" at ext2fs, du finds it:
cindy:/windows$ du --max-depth=1
19456 ./backup
.
.
1120 ./car
64 ./Media
960 ./ext2fs

The machine is running edgy. The partition is vfat, mounted without utf-8:
cindy:/windows$ mount |grep windows
/dev/hda1 on /windows type vfat (rw,umask=007,gid=46)

[Micah Cowan (micahcowan)]

Hm, while not what I was expecting to see, it still looks much more like a driver/disk problem than du's... but now I'm curious what could make du fail like that, while ls -l doesn't.

Also, the fact that "looking" at ext2fs/ with ls -l "fixes" du's ability to se ext2fs seems to support this conclusion strongly.

I'm going to set the coreutils bug back to Confirmed, out of respect for its previous setting; I'll also set the kernel bug the same. Hopefully, we'll be able to rule out one or the other, and reject it appropriately.

[Micah Cowan (micahcowan)]

Changing the linux source version to 2.6.17, as AFAICT it has not yet been reproduced on feisty.

Has fsck been run on the partition that is producing problems?

Also: are you using an ext2fs driver in windows that includes write support? I have had mild corruption issues from such things, myself.

[Jimmy P (jimmypoor2004)]

I get the same problem on feisty:
emmy:/media/sda2$ du --max-depth=1
641440 ./i386
10880 ./valueadd
40768 ./dotnetfx
3008 ./book
224 ./sysinfo
3778080 ./windows
2097088 ./Documents and Settings
4787328 ./Program Files
du: `./System Volume Information': No such file or directory
du: `./found.000': No such file or directory
du: `./Recycled': No such file or directory
.
.

emmy:~$ uname -a
Linux emmy 2.6.20-15-generic #2 SMP Sun Apr 15 07:36:31 UTC 2007 i686 GNU/Linux

emmy:~$ mount |grep sda2
/dev/sda2 on /media/sda2 type vfat (rw,utf8,umask=007,gid=46)

I don't think I've ever used an ext2fs driver in windows on this machine. The partition that displays these problems is vfat anyway.

[Micah Cowan (micahcowan)]

Setting back to 2.6.20, per feisty confirmation.

Have you run fsck on the vfat partition?

[Jimmy P (jimmypoor2004)]

emmy:~$ sudo fsck.vfat /dev/sda2
dosfsck 2.11, 12 Mar 2005, FAT32, LFN
/dev/sda2: 142476 files, 1155689/1186009 clusters

That is all the output that fsck gives. I am guessing that means the filesystem is ok?

[Jimmy P (jimmypoor2004)]

I installed mandriva 2007.1 on my machine and I get the same error.

coreutils-5.97-6mdv2007.1

[Cruncher (ubuntu-wkresse)]

I posted this problem to the coreutils mailing list and got a response.
The du from the current 6.9 version of coreutils is working nicely, so it doesn't seem to be a kernel problem.

http://www.gnu.org/software/coreutils/
(Note: coreutils 6.9 won't compile directly, due to a more recent change in glibc. Either use the current 6.9.90, or replace all occurences of "futimens" by "gl_futimens" (src/touch.c src/copy.c lib/utimens.c lib/utimens.h))

[Cruncher (ubuntu-wkresse)]

For reference, here is some detailed information:

"The underlying code that changed was in lib/fts.c.
- du, chmod, chown, chgrp, and the new chcon use fts.
- ls and cp use explicit recursion" (and are therefore not affected)

[Launchpad Janitor (janitor)]

Beginning with the Hardy Heron 8.04 development cycle, all open Ubuntu kernel bugs need to be reported against the "linux" kernel package. We are automatically migrating this bug to the new "linux" package. However, development has already began for the upcoming Intrepid Ibex 8.10 release. It would be helpful if you could test the upcoming release and verify if this is still an issue - http://www.ubuntu.com/testing . If the issue still exists, please update this report by changing the Status of the "linux" task from "Incomplete" to "New". We appreciate your patience and understanding as we make this transition. Thanks!

[Cruncher (ubuntu-wkresse)]

Since 8.04 is using coreutils 6.10, this bug appears to be gone for me. Can the people that experienced the bug check again please? Maybe this bug can be closed then (at least for Hardy and after that).

[Jimmy P (jimmypoor2004)]

seems to work fine in hardy. thanks!

[Launchpad Janitor (janitor)]

Per a decision made by the Ubuntu Kernel Team, bugs will longer be assigned to the ubuntu-kernel-team in Launchpad as part of the bug triage process. The ubuntu-kernel-team is being unassigned from this bug report. Refer to https://wiki.ubuntu.com/KernelTeamBugPolicies for more information. Thanks.

[rusivi2 (rusivi2-deactivatedaccount)]

Jimmy P:

Thank you for your contribution to this bug. If you notice this is still an issue in Mandriva, please post via their bugzilla: https://qa.mandriva.com/ Thanks!

[C de-Avillez (hggdh2)]

@rusivi1: please do not touch tasks that are not Ubuntu.