Evolution 2.24.1 really slow with big IMAP folders

Bug #292739 reported by Fabio Muzzi on 2008-11-02
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
evolution-data-server
Fix Released
Medium
evolution-data-server (Ubuntu)
Low
Ubuntu Desktop Bugs
Intrepid
Undecided
Unassigned
Jaunty
Low
Ubuntu Desktop Bugs
evolution-exchange (Ubuntu)
Low
Unassigned
Intrepid
Undecided
Unassigned
Jaunty
Low
Unassigned

Bug Description

Binary package hint: evolution

I have just upgraded from Ubuntu 8.04 to 8.10 (32 bit). The older version of Evolution worked perfectly and was really fast with my IMAP accounts (Dovecot imapd, gigabit ethernet), even if I have 4 accounts, 80 folders and 150.000 emails.

After the upgrade to ubuntu 8.10 (Evolution 2.424.1) it has become painfully slow when checking IMAP folders for new mail. It uses up to 1 GB of memory, it continuosuly accesses the disk, and it seems to hang for a lot of time (while accessing disk) on "Fetching summary information for new messages in "some folder" (100%)". The network activity is minimal and it seems "normal" to me, while disk, ram and cpu usage are very high.

The only difference in my setup is that I have upgraded from ubuntu 8.04 to 8.10.

Other useful information: The server runs Debian Etch, Dovecot is version 1.0.rc15, since I use server side filtering, Evolution is set up to check for new mail in every folder. I have no filtering enabled in Evolution.

I suppose that the issue is with the adoption of a different database format for storing folder informations.

ProblemType: Bug
Architecture: i386
DistroRelease: Ubuntu 8.10
ExecutablePath: /usr/bin/evolution
NonfreeKernelModules: nvidia
Package: evolution 2.24.1-0ubuntu2
ProcEnviron:
 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11:/usr/games
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: evolution
Uname: Linux 2.6.27-7-generic i686

Fabio Muzzi (kurgan-kurgan) wrote :
Sebastien Bacher (seb128) wrote :

thank you for your bug report, the issue is an upstream one though, could you open the bug on bugzilla.gnome.org where the people writting the software will read it? optimization work is rather upstream work than distribution one, especially that you are the first one to complain about the new version speed so some option you are using could trigger the issue

Changed in evolution:
assignee: nobody → desktop-bugs
importance: Undecided → Low
Fabio Muzzi (kurgan-kurgan) wrote :

It seems that someone has already reported it upstream, this is the link to the upstream bug report:

http://bugzilla.gnome.org/show_bug.cgi?id=558883

Hope this helps.

Changed in evolution:
status: New → Triaged
Fabio Muzzi (kurgan-kurgan) wrote :

As I can see, patches are already being developed upstream.

I hope they will be soon included in Ubuntu, and I hope that they will be pushed as updates as soon as possibile, because I can't afford to wait for Ubuntu 9.04 to have a working Evolution again.

Sebastien Bacher (seb128) wrote :

did you try the upstream changes? can you confirm they fix your issue?

Fabio Muzzi (kurgan-kurgan) wrote :

I have manually created indexes as stated in the upstream bug report, I have not recompiled Evolution, so I don't know if the patch actually works.

Anyway, even if I have created indexes, Evolution is still slow (not as much as before) because (as it seems from upstream discussion) it still handles lots of UIDS using a slow method. (see post #8)

I think that if this issue is not solved upstream properly and in a reasonable time, it could be nice to have the old version of Evolution packaged for Ubuntu 8.10, so people can install the old one if they need.

Switching to a new database format seems to have created a lot of efficiency issues, that will probably need months to solve upstream, so maybe packing the old Evolution for Intrepid is the way to go.

Thanks for your prompt answers and continued help.

Sebastien Bacher (seb128) wrote :

the old version will not be distributed in intrepid no since that would be lot of extra work for almost no win, you can still use the lts version if you need to run the previous version

Vasil Kolev (vasil-ludost) wrote :

I tried the trick with creating indexes and for me evolution now is even faster than the previous version. I'm using it on reiserfs, a normal intrepid upgrade on x86_64.

Fabio Muzzi (kurgan-kurgan) wrote :

Upstream analisys continues.

http://bugzilla.gnome.org/show_bug.cgi?id=558883

It seems that the bug is far more evident (Evolution is really slower) when you have more than one account. There is some sort of concurrency issue.

Mario Manno (manno) wrote :

I applied both patches to e-d-s, the index patch really helps a lot.
If you want to test yourself you can get packages at my ppa http://ppa.launchpad.net/manno/ubuntu

Fabio Muzzi (kurgan-kurgan) wrote :

I have tried the updated version, and it is still terrilbly slow when you have more than one IMAP account. If I keep only one account enabled and disable all the others, it becomes notably faster. I suppose that there is some sort of concurrency issue that forces continuous flushes to disk to the sqlite3 database files.

This issue has been reported by me at http://bugzilla.gnome.org/show_bug.cgi?id=558883#c23

Vasil Kolev (vasil-ludost) wrote :

Fabio, do you use filters in Evolution to sort the email? I found out that if I disable filtering on the INBOX (while still having two IMAP accounts) it's definitely better. Seems like it's filtering the whole INBOX (there are about 3k messages there) instead of just the new stuff.

Fabio Muzzi (kurgan-kurgan) wrote :

I use server-side filtering, so I have completely disabled filtering in Evolution. But I have enabled checking for new mail in every folder, since I get mail delivered directly to the folders by the server.

If you look at the upstream bug tracking page, at http://bugzilla.gnome.org/show_bug.cgi?id=558883, you'll see that there are new patches addressing issues like the one you seem to have. Basically it seems that Evolution is rescanning every single email in every single folder when checking for new mail, and maybe it does the same when filtering.

I am really disapponted by the incredible number of crippling bugs that this update of Evolution has introduced.

Fabio Muzzi (kurgan-kurgan) wrote :

It seems that patches have been commited to stable tree upstream.

http://bugzilla.gnome.org/show_bug.cgi?id=558883#c45

Maybe it's time to include these patches and post the new binary packages to proposed upates.

Sebastien Bacher (seb128) wrote :

new GNOME tarballs are due today so there is no need to hurry backporting patches now rather than using those

Sebastien Bacher (seb128) wrote :
Changed in evolution-exchange:
importance: Undecided → Low
status: New → Confirmed
Martin Pitt (pitti) wrote :

Accepted evolution-exchange into intrepid-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in evolution-exchange:
status: New → Fix Committed
Changed in evolution-data-server:
status: New → Fix Committed
Martin Pitt (pitti) wrote :

Accepted evolution-data-server into intrepid-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Fabio Muzzi (kurgan-kurgan) wrote :

I have just installed the new evolution-data-server from proposed, it is considerably faster than before, but still not as fast as the old evolution that did not use sqlite. It seems to me that when it runs a lot of concurrent activities (like scanning different folders for different accounts for new messages) it slows down a lot.

If I enable 4 accounts, a full "send/receive" runs for one minute (with gigabit connection to the imap server). If I enable only one account, a full "send/receive" runs for 4 seconds with quite no disk activity. It is obvious that there is something wrong when Evolution runs imap folder update tasks concurrently.

It also uses a lot of memory, and grows indefinitely: after 10 minutes it's using more than 400 MB of memory.

Martin Pitt (pitti) wrote :

Thanks Fabio for testing. So it's still not "good" yet, but since it is an improvement, I mark this as verified, so that this version can go to -updates.

Fabio Muzzi (kurgan-kurgan) wrote :

I agree, it's an improvement.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package evolution-data-server - 2.24.2-0ubuntu1

---------------
evolution-data-server (2.24.2-0ubuntu1) intrepid-proposed; urgency=low

  * New upstream version:
    Bug Fixes:
    - #546637: Now unread vfolder should be better (lp: #275952)
    - #557348: Clear summary when we reset query.
    - #558883: Improve performance in case of large IMAP folders. (lp: #292739)
    - #560076: Handle followup better
    - #561069: If nothing to expunge, don't crash. (lp: #298526)

evolution-data-server (2.24.1.1-0ubuntu1) intrepid-proposed; urgency=low

  * New upstream version
    Bug Fixes:
    - #209514: (Novell Bugzilla) Evolution Groupwise missing mails
    - #372382: (Novell Bugzilla) VUL-0: OTRS SOAP interface vulnerability
    - #434946: (Novell Bugzilla) Deleted Messages in Trash Show Bold and Behave
      Like New Messages
    - #434950: Newly Fetched Mail Doesn't Update Mailbox Count in
      QuickViewer Until Exit and Entry
    - #434958: 'Empty Trash' Sometimes Requires Multiple Attempts Before
      Initiating
    - #435725: Disable Right-Mouse "Delete" On GrouPWise System Email Folders
    - #435727: Deleted Shared Folder Crashes Evolution
    - #435964: Evo crashed after switching from offline to online
      in GW addressbook
    - #440502: Tracker Bug: GroupWise Magic Patch Tracker
    - #532136: Evolution crashed - modifying a local task and changed offline
      state
    - #555979: Evolution does not empty trash on exit (lp: #281093)
    - #556119: Takes a very long time to index newsgroup.
    - #558727: Format string bugs causing potential crashes
    - #558737: Evolution freezes on trying to store folder
    - #558883: Evolution 2.24 is terribly slow with large IMAP folders.

 -- Sebastien Bacher <email address hidden> Mon, 24 Nov 2008 15:09:05 +0100

Changed in evolution-data-server:
status: Fix Committed → Fix Released
Martin Pitt (pitti) wrote :

Copied e-d-s intrepid-proposed to jaunty.

Changed in evolution-data-server:
status: Triaged → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package evolution-exchange - 2.24.2-0ubuntu1

---------------
evolution-exchange (2.24.2-0ubuntu1) intrepid-proposed; urgency=low

  * New upstream version:
    Bug Fixes:
    - #540346: Do not try to fetch contacts when not connected
    - #558883: Improve performance in case of large IMAP folders (lp: #292739)

 -- Sebastien Bacher <email address hidden> Mon, 24 Nov 2008 15:18:45 +0100

Changed in evolution-exchange:
status: Fix Committed → Fix Released
Martin Pitt (pitti) wrote :

e-exchange intrepid-proposed copied to jaunty.

Changed in evolution-exchange:
status: Confirmed → Fix Released
Mario Manno (manno) wrote :

Please note the upstream bug is still open.
However the latest (4th December) patch does away with dbsort and this helps a lot. You can try my ppa if you want to test it.

Fabio Muzzi (kurgan-kurgan) wrote :

Mario, I'm testing your patched Evolution now. It still has a bug regarding update of unread messages count in folders pane, which makes it hard to tell if I have got new mail or not. (see http://bugzilla.gnome.org/show_bug.cgi?id=559391)

It seems to be still slow compared to the version that used the old database backend, but it now seems at least acceptable.

I'll test it further, while at the office (with a GB connection to the imap server) and out (with a 512K adsl uplink).

Changed in evolution-data-server:
status: Unknown → Confirmed
Steve Langasek (vorlon) wrote :

Accepted into intrepid-proposed, please test and give feedback here. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Vasil Kolev (vasil-ludost) wrote :

Using the version from proposed, it's definitely a bit better, but still with two reasonably big IMAP accounts my drive grinds for a few minutes until everything is updated. It's still worse than with hardy, but somewhat livable.

Vasil Kolev (vasil-ludost) wrote :

Okay, I finally used my head for a while, and found a work-around for this problem. In short,
cat ~/evolution/mai/imap/*/folders.db > /dev/null
before starting evolution seems to make it load at least as fast as the versions before sqlite.

Now, this seems to be an issue either with the block device scheduler, sqlite or both. It's visible that when you have two tasks that do a lot of disk IO (firefox tends to do that when first reading the location bar cache, or updatedb, or dpkg), then the whole system grinds down to a halt. Evolution does the same just by itself, by doing a lot of small reads on two files in parallel and most drives don't really like that.
(I might try testing this on a RAID1 array of faster drives, if that could be useful)

For the record, I'm using standard intrepid + proposed.

Fabio Muzzi (kurgan-kurgan) wrote :

Dear Vasil,

There are discussions going on elsewhere about the poor performance of the block device scheduler in new kernels, so maybe you are right. Evolution does a lot of small i/o requests, and the whole problem gets really worse when you have more than one account, because the number of parallel requests grows with the number of accounts you have.

Maybe this performance problem is caused by a sum of causes: poor coding, poor sqlite performance, poor kernel i/o scheduler performance. We should test by recompiling the kernel (on the same hardware) and changing the i/o scheduler it uses.

Vasil Kolev (vasil-ludost) wrote :

Fabio,

Changing the IO scheduler is easy - it's in /sys/block/$devname/queue/scheduler ($devname = sda in my case), and ubuntu comes with all the normal in-kernel schedulers compiled in - deadline, anticipatory, noop and cfq (the last one being the default one).

I'll try this with the other schedulers and will see if that makes any difference. BTW, can you post some links to the discussions about the block scheduler performance, they might be useful to this bug too.

I don't think that SQLite is the problem, as when I preload it it's blindingly fast - and when you have a single account, it's again really fast. If someone can test evolution with postgresql or mysql, I think we'll see the same results (without the preloading).

Fabio Muzzi (kurgan-kurgan) wrote :

Vasil,

I have tried testing every scheduler available. I have copied a 700 MB file between tests to flush the cache.

The results are as follows:

- Noop is the slowest scheduler, by maybe 5-10%, no more.

- All of the other schedulers seem to have the same performance.

- If I don't flush the cache, Evolution is *REALLY* fast (at least 5 times faster, maybe 10) with every scheduler.

- Just for information, copying the 700 MB file from and to the same disk takes approx 36 seconds with every scheduler but "noop", and 40 with "noop".

So the bottom line, at least on my system (Dell Precision M65 laptop, 1 GB RAM, 7200rpm sata hdd, 32 bit Core Duo T2500, seems to be that basically all schedulers show equal performance.

I have not tried to see how responsive is the system while running the tests, so my test does not focus on multitasking or desktop responsiveness, only on the timing of a single operation that basically can use all of the CPU time and disk I/O bandwiidth (copying a single file or opening Evolution).

Changed in evolution-data-server:
status: Confirmed → Fix Released
Changed in evolution-data-server:
importance: Unknown → Medium
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.