Baobab reports incorrect sizes

[JohnLM]

Binary package hint: gnome-utils

I've noticed the baobab (aka Disk Usage Analyzer) is not reporting sizes of directories correctly.
I my particular case all scanned subfolders are readable (and even writable) for user, so that is not a problem.

After a few tests I concluded bug is within the gnome-utils package of versions 2.24.1 and up. I tried compiling number of versions on the same system, and it showed that older (2.20.0.1) have no such bug, only later versions. (A regression bug?)
Anyway bug in dependent libraries is unlikely, cause in test all versions were linked against the same external libraries.

There are bunch of screenshots added showing the bug.
Thanks for Ernst's research it was pointed that it is NTFS long names that interferes with Baobab's scanning module. (see comments and links below for details)
Also on ext3 bug somewhat showed itself, though with different reason obviously. Still Paolo's patch reportedly takes care with that as well.

Thanks to Paolo for developing patch!
Until patched release is out, you'll have to patch and compile from source.

Affected:
gnome-utils baobab 2.24.1 up to 2.26.0 w/o patch
on all tested platforms (those currently being x86 and x86_64)

[Fabio Marzocca (thesaltydog)]

We can investigate more, but I can assure that baobab scans each file in each directory, without discarding anything that is readable to the user.
Don't know what nautilus does. Could you pls check if your "misc" folder contains any not standard file?

[JohnLM (johnlm)]

No I believe it doesn't contain any non-standard file.

btw If it changes anything, partition is ntfs (fuse-ntfs3g), as you may have guessed.
One thing what that suggests me nautilus is right is that Windows reports the same numbers!

Also when I said I think it takes largest sub-folders... I in fact took a sum of those folders from pie chart on calculator.
They equal the baobab's reported size, but there is actually more within the "misc" folder.
And heck! 227 GiB vs. 322 GiB (327 - 5 free) is nearly 100 GiB difference, that ain't a small thing.
With all due respect, there must be a bug or something in code!
I know little programming, so I might take a peek into code myself, but I am not familiar with baobab's code, so it would make a tiny difference.

Now just to be clear, I might try copying the folder on ext3 partition, and see what happens.

Oh Yes, I'm glad I got a response so soon! Thanks at least for that, yet!

[Fabio Marzocca (thesaltydog)]

Maybe you are displaying allocated space. Try switching the checkbox in View->Allocated Space

[JohnLM (johnlm)]

Tried switching 'Allocated Space'. Changes by quite tiny numbers... nowhere close difference I'm having.

btw also tried on ext3... does the same thing.
As I said it apparently this happens with folders with large number of files in several levels of sub-folders. (At least it is easily noticeable for these)

also forgot to mention I'm using x86-64 system. (I don't believe it changes anything, though)

I might later try various LiveCDs with baobab (both x86 and x86-64).
I might also try upstream version of baobab.
However this bug report is still relevant to this particular version. (being Baobab 2.24.1 in this case)

[JohnLM (johnlm)]

Well I tried baobab (2.20.0.1) on Hardy's LiveCD. Works fine... I guess you can consider this a regression bug.

[Fabio Marzocca (thesaltydog)]

That' very strange, as no change has been made in the scanning module from Hardy to Intrepid...

[JohnLM (johnlm)]

Strange indeed. I doubt there is something radically different between Hardy's and Interpid's enviroments! Err... except me using 64bit Interpid and having 32bit Hardy LiveCD. I think I should also download Interpid 32bit (x86), just to be sure it is not on 64bit only.
I thought there could be a change in dependent packages. However those are largely shared with nautilus.

There is however libgnomevfs2-0 (used by gnome-utils) and gvfs-backends (used by nautilus), what seems to be two packages with the same apparent purpose.
I'm really not sure if this is relevant at all, but it is only lead I have so far.

[Ernst (ernst-blaauw)]

Hi, I'm also affected by this bug. See the attached screenshot:
- GParted shows 40 GB is used
- The 'properties' window of Nautilus shows 40 GB used
- However, Disk Usage Analyser shows only 2.7 GB.

I'm using 8.10, 32-bit, with the backports and proposed repo's enabled.

[JohnLM (johnlm)]

So I did some little tests. From now on this is definitely NOT a distro or architecture specific bug!

Bug is present:
On both Interpid x86 and x86_64, and also on Fedora 10 x86_64 (and probably x86 as well)
Fedora is also using Baobab 2.24.1

There is also one more thing I've noticed... it may not be really useful, but still
After scanning a folder with baobab it's size is cached in memory (which is usual), though it's not the full size.
Then when opening nautilus properties window, it doesn't start counting size from 0, but from cached size, up to actual size.

I guess that means both use same libraries (and the same cache) and there is some files baobab overlooks, which nautilus just adds up. So really, I think if it is nothing else then it should be gnomevfs thing... I can't think of anything else now.

Also need to check how deep in directory tree baobab traverse.

[Fabio Marzocca (thesaltydog)]

Baobab doesn't use gnome-vfs. It uses (as nautilus does) Gio.

[Fabio Marzocca (thesaltydog)]

I don't know how nautilus sums up space in the Properties window, but Baobab makes a full scan, file by file, traversing every directory and getting the size from g_file_info_get_size() from Gio library.

[JohnLM (johnlm)]

It uses Gio? Might have missed that cause it doesn't show up as direct dependency in APT.
And it does scan file by file, hmmm...
Anyways, this just makes it all the more cryptic...

I might try to play with the code, though I'm no expert.
I'll post if I have any luck!

[JohnLM (johnlm)]

OK I did a bit more tests.
I didn't look at the actual code yet, but I grabbed three gnome-utils tarballs from gnome.org and built them.
So I built them on my already mentioned Interpid box within home folder (so I don't break preexisting gnome-utils).
Then run both compiled and repository versions side by side to see what they do.
Results:
2.20.0.1 does everything right!!!
2.24.1 behaves exactly as baobab from repository package.
2.26.0 (currently upstream) does exactly the same thing as 2.24.1

What I could learn that bug is indeed within gnome-utils package, cause I built all of them against the same developer libraries.
I also added three screenshots to see how it looks on screen.

Well for those affected by this bug... my current recommendation would be to revert to 2.20.0.1 (simply install Hardy's package or compile from source).

[Ernst (ernst-blaauw)]

I reported this bug earlier on 8.10. I can confirm this bug on 9.04 (Jaunty) 32-bit too. (I reinstalled, no dist-upgrade.) Maybe it is useful to report that this bug appears on my ntfs share (as you can see in the screenshot I posted earlier).

[Fabio Marzocca (thesaltydog)]

Unfortunatley, I have no ntfs shares available at the moment. I have tested this functionality on a remote ssh and ftp site, and it is working correctly.

[JohnLM (johnlm)]

Try testing it on local filesystem (like ext3).
Make sure you have a lot of files and lots of subfolders so they make up the difference.

[Fabio Marzocca (thesaltydog)]

As I reported in a previous post, Baobab gets the size with g_file_info_get_size() from Gio library. Nothing fancy. Maybe a Gio bug? I will investigate with developers.

[Ernst (ernst-blaauw)]

If I do

du -h --max-depth=0

inside the 'problematic' map, I get the output:

21G

Baobab tells me the size of the folder is 1.0G.

If you want more tests, please ask me: I really want to help to solve this bug!

[Fabio Marzocca (thesaltydog)]

 @Ernst

Thank you for reporting, but I still can reproduce the bug. It could be interesting to investigate why this folder is so "problematic"...

Do you have soft or hard links inside that folder? Something else not standard?

[Fabio Marzocca (thesaltydog)]

@Ernst

Do you get any difference by switching the View->Allocated Space on and off?

[Ernst (ernst-blaauw)]

Switching the view doesn't matter.

However, Baobab reports I have hard links in the folders - after expanding the folders, I think it accounts for all the 'lost' space inside Baobab.

Thus, I went to a folder which showed empty and remove a character. Now, Baobab show this file in the summary. If I added the character back to the file name, the file was still shown in Baobab. (That's strange, isn't it?)

I went to other subfolders and a 'ls -li' showed the files had 2 hard links. However, I cannot remember I created those hard links. And besides - renaming shoudn't remove hard links. Is it possible this is a bug in ntfs-3g?

But back to this bug report: Is it a normal behavior Baobab doesn't show hard links? As all file entry's are actually hard links, I think Baobab should just count the size of a hard link.

[Ernst (ernst-blaauw)]

I searched for 'ntfs-3g hard links' and I found this forum thread: http://forum.ntfs-3g.org/viewtopic.php?f=2&t=1149
It is actually about Baobab and gives a reason why some files have multiple hard links in NTFS.

Hopefully, this is useful to you :-)

[Fabio Marzocca (thesaltydog)]

As I am understanding, this possibile bug is related only to ntfs long names, so maybe the title should be tuned up.

Anyway, hard links are correclty reported in Baobab. Try scanning /usr/bin folder and you will see that in the folder description will be displayed "Contains hard links for: xx MB".

Concerning ntfs-3g, I can just repeat that we are using standard Gnome Gio libraries, so I have to check with them if they have any report on this.

[Ernst (ernst-blaauw)]

What do you mean by 'hard links are correctly reported'? In my case, files with more than one hard link are not used in the ring chart and in the overview of sizes (which is not correct).

So, Baobab gets the (correct) information about the file sizes. However, if a file has more than one hard link, Baobab does not count the size of a file. It states only that there are hard links (with correct size), but discards the file in the total calculation. In my opinion, all files have hard links (most files have one hard link) and should just be counted.

Furthermore I have 15 kb of hard links in my /usr/bin. As they are only 15 kb, I cannot see if they are actually counted.

I think the bug is inside Baobab: The way (more than one) hard link(s) is/are processed.

Thanks for the time you take to help with this bug :-). Hopefully, my post is helping to solve the solution. Please tell me if I talk nonsense ;-).

[Fabio Marzocca (thesaltydog)]

You are welcome, Ernst. Thank you.

[JohnLM (johnlm)]

hmm took a peek into code... can't say I understand much, but I noticed version 2.24.1 uses Gio... just as you said, however 2.20.0.1 is using gnome-vfs - that is quite a change in scanning module.
on other hand, a quick thought of mine was that we concentrate on scanning module while the bug might as well be in interface code (nothing to support this, but we wouldn't want to overlook anything)

btw sorry but I'm off bughunting for now... installed new distro (no gnome)

[Fabio Marzocca (thesaltydog)]

@JohnLM: I'm not sure it could be related to the interface code. That part just displays a total amount number.

[Bill Morgan (therealbrewer)]

I can confirm this bug. I've got several external USB drives formatted as NTFS. These drives were filled with media files over many older versions of Ubuntu, though now I'm running Jaunty. When Baobab scans these drives, it reports a wrong size for almost every folder containing files with long names. It reports all these folders as "contains hardlinks for: XX GB". However, I have discovered a (tedious) workaround: if I simply move all the files in that folder to a temp folder, then move them back, Baobab reports the folder size correctly (to the first level). However, this does not work recursively down the tree, so as you can imagine, this can be quite cumbersome if the tree structure is complicated. The workaround works whether I move the files using nautilus or with a terminal. Since I have a large media collection and a complex tree structure, I would like to automate the workaround so I don't have to touch every file by hand. I have been able to incorporate the workaround into a script, but it doesn't work recursively yet. Oh, and speaking of touching the files, "touch *" is not enough, nor is it enough to simply rename the folder. I don't know why, but I have found that the files have to be moved to a new folder, then back.

[Paolo Borelli (pborelli)]

Can you guys try with this patch and see if it improves things?

Even if it doesn't fix things, it would be helpful to know if the warning "Could not obtain inode and device for hardlink" is printed to the terminal

[Paolo Borelli (pborelli)]

ops, sorry attached the wrong version of the patch, here is one that actually compiles.

(patch is against latest git version, but should apply also to previous versions)

[Ernst (ernst-blaauw)]

I'd love to test this out, but I never used a patch. I will check out the
repository with the latest code, but how do I apply the patch to the source
code?

On Sun, Jun 28, 2009 at 13:41, Paolo Borelli <email address hidden> wrote:

> ops, sorry attached the wrong version of the patch, here is one that
> actually compiles.
>
> (patch is against latest git version, but should apply also to previous
> versions)
>
> ** Attachment added:
> "0001-Fix-gio-inode-and-device-handling-for-hardlinks.patch"
>
> http://launchpadlibrarian.net/28481108/0001-Fix-gio-inode-and-device-handling-for-hardlinks.patch
>
> --
> Baobab reports incorrect sizes
> https://bugs.launchpad.net/bugs/341141
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[Paolo Borelli (pborelli)]

get the sources either from git or from a released tarball and then apply the patch with

patch -p1 < 0001-Fix-gio-inode-and-device-handling-for-hardlinks.patch

[Ernst (ernst-blaauw)]

This patch fixed it for me in the gnome-utils-2.26.0 package :-).
Thanks!

On Sun, Jun 28, 2009 at 14:29, Paolo Borelli <email address hidden> wrote:

> patch -p1 < 0001-Fix-gio-inode-and-device-handling-for-hardlinks.patch
>

[Fabio Marzocca (thesaltydog)]

Patch committed to current development branch. Will be available in next major release.

Thanks Paolo!

[Bill Morgan (therealbrewer)]

That worked for me too.
Thanks!

[Onlyodin (ubuntu-xsnet)]

Yep, that worked a treat, thanks Paolo :-)

[Onlyodin (ubuntu-xsnet)]

JohnLM, I was experiencing that with ext3 as well, the problem wasn't just NTFS related.

[Paolo Borelli (pborelli)]

Thanks everybody for testing.

Actually the patch had a small bug in it and I committed a small modification on top of that patch:

http://git.gnome.org/cgit/gnome-utils/diff/baobab/src/baobab-scan.c?h=gnome-2-26&id=bc09d54a97723fc524b23c20b9a2bbb81524afcd

Can you guys test if things still work with this additional fix?

[JohnLM (johnlm)]

@Onlyodin
Was it? Well I also had this with ext3, but disregarded that as copy-paste from ntfs.
oh well... more description edits...

[Ernst (ernst-blaauw)]

If I copy paste this into a file and do patch -p1 < patch_file (in my
already patched gnome-utils-2.26.0), I get:

patching file baobab/src/baobab-scan.c
patch: **** malformed patch at line 6: hl.device =
g_file_info_get_attribute_uint32 (s,

How should I apply this one?
Ernst

On Mon, Jun 29, 2009 at 16:15, Paolo Borelli <email address hidden> wrote:

> Thanks everybody for testing.
>
> Actually the patch had a small bug in it and I committed a small
> modification on top of that patch:
>
> http://git.gnome.org/cgit/gnome-utils/diff/baobab/src/baobab-
> scan.c?h=gnome-2-26&id=bc09d54a97723fc524b23c20b9a2bbb81524afcd<http://git.gnome.org/cgit/gnome-utils/diff/baobab/src/baobab-%0Ascan.c?h=gnome-2-26&id=bc09d54a97723fc524b23c20b9a2bbb81524afcd>
>
> Can you guys test if things still work with this additional fix?
>
> --
> Baobab reports incorrect sizes (with NTFS long-name files)
> https://bugs.launchpad.net/bugs/341141
> You received this bug notification because you are a direct subscriber
> of the bug.
>

[Paolo Borelli (pborelli)]

Sorry, here is a properly formattaed patch

[Bill Morgan (therealbrewer)]

The typo patch works fine.

[Onlyodin (ubuntu-xsnet)]

Yep, all good with the typo fixed for me too.

[Alex (dinesh-email)]

i tried to patch the source gnome-utils_2.26.0.orig.tar.gz

I received the error:

patching file baobab/src/baobab-scan.c
Hunk #1 FAILED at 102.
1 out of 1 hunk FAILED -- saving rejects to file baobab/src/baobab-scan.c.rej

where did i nade mistakes?

thank you for the answer.
Alex

[Onlyodin (ubuntu-xsnet)]

Alex, try applying the ubuntu patch found here:

https://launchpad.net/ubuntu/jaunty/+source/gnome-utils/2.26.0-0ubuntu1

Then apply the patches further up this thread. Worked for me.

[Alex (dinesh-email)]

i downloaded the source code from there but the patch

http://git.gnome.org/cgit/gnome-utils/diff/baobab/src/baobab-scan.c?h=gnome-2-26&id=bc09d54a97723fc524b23c20b9a2bbb81524afcd

return the error i posted.

the other two files (***.diff.gz and ***.dsc) aren't patches ... am I wrong?

Thank you for answering
Alex

[Onlyodin (ubuntu-xsnet)]

Alex, the .diff.gz file is the one you want, as the .gz extension denotes, this file is gzipped and needs to be decompressed prior to patching the original source.

You then need to apply the first patch Paolo submitted: http://launchpadlibrarian.net/28481108/0001-Fix-gio-inode-and-device-handling-for-hardlinks.patch

prior to applying the second (which is intended to correct a change made by Paolo's first patch): http://launchpadlibrarian.net/28511243/0001-Fix-typo-in-the-last-patch.patch

[Alex (dinesh-email)]

Onlyodin,

as you can figure is the first time i compile from source. I'm pretty newbe at this level. Thank you for your message.

I applied all the patches in the right sequence and I successfully complied the package.
Unfortunately running baobab returns many critical errors. Maybe i have the errors because I'm using ubuntu 8.10 and not 9.04.

thank you for your support!

[David Balažic (xerces8)]

Is this fixed in Ubuntu 9.10 "Karmic Koala" ?

Or is there any chance for a SRU for 9.04 ?
It does not affect any critical infrastructure packages and the patch is a few lines.

Because so baobab is pretty useless, as it show completely wrong data for ntfs (and less frequently for ext3).

[Gustav Svensson (gustav-svensson)]

Thanks for the fix! Working fine now on Jaunty x86_64 patching gnome-utils-2.26.0 with the two patches above.

[Łukasz Czerwiński (milimetr88)]

I have a similar problem in baobab 2.30.0.
I started baobab from root terminal to be sure that all directories are counted. But it seems that instead of counting all directories, some of subdirectories from my /home/lukasz directory are ignored! (see attachment)

[Wiley (coyotewilee99)]

I have the same problem as Lukasz in Lucid with baobab 2.30.0.

[JohnLM (johnlm)]

Hmm... interesting. Previous tests never shown any difference between user and root.
Make sure scanned folders and files are user-readable, and both times you use the same "Apparent size" setting.
If bug still presents itself, try patching baobab and try again. (It appears though you can patch only 2.26.0... I haven't tested)