Heavy disk I/O usage when starting KMail (reiserfs)

[Daniel Hahler]

Binary package hint: kdepim

It appears that since I've upgraded to kmail 4:3.5.7enterprise20070810-0ubuntu1 (in Gutsy), launching KMail for the first time causes heavy disk usage, for several minutes.
My guess is, that this is related to scanning all the mails in my "Local folders" somehow, where I have ~95000 mails.

Using "lsof" it shows that there are two kio_file processes on directories in ~/.kde/share/apps/kmail/mail/.

I have not experienced this before the upgrade to the "enterprise" packages.

[Thomas Templin (coastgnu)]

Same behaviour here.
Also a lot of mails ~5GByte

[Daniel Hahler (blueyed)]

Still the case with 4:3.5.7enterprise20070904-0ubuntu2.

Here's a backtrace, for the two kio_file processes, which cause this freezing behaviour:
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb7dce6d3 in __old__lxstat64 () from /lib/tls/i686/cmov/libc.so.6
#2 0xb672d76a in FileProtocol::createUDSEntry (this=0xbfe739d4, filename=@0xbfe72784, path=@0xbfe7273c, entry=@0xbfe7274c, details=2, withACL=true) at /usr/include/sys/stat.h:509
#3 0xb6730a47 in FileProtocol::listDir (this=0xbfe739d4, url=@0xbfe7381c) at /build/buildd/kdelibs-3.5.7/./kioslave/file/file.cc:1251
#4 0xb7b3e422 in KIO::SlaveBase::dispatch (this=0xbfe739fc, command=71, data=@0xbfe73978) at /build/buildd/kdelibs-3.5.7/./kio/kio/slavebase.cpp:1061
#5 0xb7b45bbd in KIO::SlaveBase::dispatchLoop (this=0xbfe739fc) at /build/buildd/kdelibs-3.5.7/./kio/kio/slavebase.cpp:293
#6 0xb672bba8 in kdemain (argc=4, argv=0x80a8610) at /build/buildd/kdelibs-3.5.7/./kioslave/file/file.cc:127
#7 0x0804e6bf in launch (argc=4, _name=0x80a789c "kio_file", args=0x80a7907 "", cwd=0x0, envc=0, envs=0x80a790c "", reset_env=false, tty=0x0, avoid_loops=false, startup_id_str=0x805102e "0")
    at /build/buildd/kdelibs-3.5.7/./kinit/kinit.cpp:673
#8 0x0804ef4f in handle_launcher_request (sock=9) at /build/buildd/kdelibs-3.5.7/./kinit/kinit.cpp:1240
#9 0x0804f328 in handle_requests (waitForPid=0) at /build/buildd/kdelibs-3.5.7/./kinit/kinit.cpp:1443
#10 0x080505bc in main (argc=5, argv=0xbfe74144, envp=0xbfe7415c) at /build/buildd/kdelibs-3.5.7/./kinit/kinit.cpp:1909
#11 0xb7d22050 in __libc_start_main () from /lib/tls/i686/cmov/libc.so.6
#12 0x0804bb71 in _start ()

[Aaron C. de Bruyn (darkpixel2k)]

Are you by any chance using IMAP?

[Aaron C. de Bruyn (darkpixel2k)]

Sorry--are you by any chance using IMAP on your local machine?
More specifically, using IMAP on your local machine, and something like fetchmail to retrieve your messages, then retrieve them from your local machine with kmail?

[Daniel Hahler (blueyed)]

No, I'm not using IMAP locally.
I have some IMAP accounts though, but they are on a remote server and I'm not using fetchmail in any way.

I'm using the local folders only for storage and some folder in there to retrieve mails from a POP3 account.

I've now temporarily moved the ~/.kde/share/apps/kmail/mail folder away and an error came up that the POP3 account has no "incoming" folder anymore.
Then I've moved the "mail" folder back and configured the incoming folder again.
I've noticed, that the folderlist in the pop3 account setup is a bit weird (see attached screenshot), maybe this is related?

I've not experienced the problem now, after having moved the "mail" folder away temporarily, but as far as I know there's some timeout/caching involved anyway: it did not happen before, when I've restarted KMail, but only after KMail has not been running for x hours/minutes.

[Lars Knoll (lars-trolltech)]

For me this doesn't only happen when starting kmail for the first time. I am using the cached imap client in kmail. since the upgrade to the enterprise branch kmail has basically become unusable for me.

The scan it does takes (a) forever and (b) basically locks up my hard disk. All other operations (like just running a simple command as top from the shell) take at least 10-20 secs to react. It might be related to my file system being reiserfs.

strace'ing one of the kio_file processes spawned by kmail shows that it seems to ask for posix acl attributes for every mail I have:

...
getxattr("1188425724.27265.vN8Kd:2,S", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
getxattr("1188425724.27265.vN8Kd:2,S", "system.posix_acl_access", 0xbf88d8e0, 132) = -1 EOPNOTSUPP (Operation not supported)
lstat64("1188425038.27265.Deu4J:2,RS", {st_mode=S_IFREG|0644, st_size=2102, ...}) = 0
getxattr("1188425038.27265.Deu4J:2,RS", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
getxattr("1188425038.27265.Deu4J:2,RS", "system.posix_acl_access", 0xbf88d8e0, 132) = -1 EOPNOTSUPP (Operation not supported)
lstat64("1188425371.27265.4Ellm:2,S", {st_mode=S_IFREG|0644, st_size=1073, ...}) = 0
.....

reiserfs seems to fail on that operation, but the whole thing seems to keep my file system completely locked up. I also do not understand at all while kio_file should ask for extended file attributes/posix acl's at all.

This is a complete showstopper for me when using gutsy at the moment.

[Daniel Hahler (blueyed)]

I'm using reiserfs, too. This might be a pattern.

What about you, coastgnu?

[Lars Knoll (lars-trolltech)]

Another observation: Adding 'acl,user_xattr' to the mount options of my home partition fixed most of the issues. Mail syncing is still not lightning fast, but it's become usable now. And most importantly: It doesn't lock up my file system/hard disk any more.

So to me this looks mostly like a bug in the reiserfs code in the kernel (even though I also see quite some potential to optimise kmail here).

[Luka Renko (lure)]

OK, nice to know there is sensible workaround.

I have discussed with upstream and they are aware or the issue and will look into it:
[11:13] <Lure> till: does enterprise branch do anything special in regards to posix acl? we have some performance issue reports on reiserfs and would be good to understand how does enterprise change this
[11:13] <Lure> till: https://launchpad.net/bugs/135147
[11:16] <till> I've seen that, yes.
[11:16] <till> Could be a kdelibs bug, I'll have a look.
[11:16] <till> I incidently wrote the acl stuff as well, in kdelibs. ;)
[11:16] <till> We have a deadline though, atm, so I won't get to it today, likely.

[Daniel Hahler (blueyed)]

Works for me in KDE 4.3.