Some emails are not displayed but are saved in the mailbox file

[mstuermer]

Binary package hint: evolution

Since yesterday's update (2008-07-09) a few but not all new emails are not displayed in Evolution. When I realized there were missing emails I looked into the mailbox file of my Inbox ("tail -n 1000 .evolution/mail/local/Inbox") and found that the emails were downloaded but not displayed! That's really very bad and made me switch to Thunderbird after over 3 years of more or less happy using Evolution.

evolution:
  Installiert:2.22.3.1-0ubuntu1
  Mögliche Pakete:2.22.3.1-0ubuntu1
  Versions-Tabelle:
 *** 2.22.3.1-0ubuntu1 0
        500 http://ch.archive.ubuntu.com hardy-proposed/main Packages
        100 /var/lib/dpkg/status
     2.22.2-0ubuntu2 0
        500 http://ch.archive.ubuntu.com hardy-updates/main Packages
     2.22.2-0ubuntu1.2 0
        500 http://security.ubuntu.com hardy-security/main Packages
     2.22.1-0ubuntu3 0
        500 http://ch.archive.ubuntu.com hardy/main Packages

[mstuermer (matthias-stuermer)]

By the way I confirmed this Evolution bug at the computer of my wife who experienced the same problem receiving important emails which were not displayed in the Inbox.

[C de-Avillez (hggdh2)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Unfortunately we can't fix it, because your description didn't include enough information. You may find it helpful to read "How to report bugs effectively" [WWW] http://www.chiark.greenend.org.uk/~sgtatham/bugs.html. We'd be grateful if you would then provide a more complete description of the problem.

We have instructions on debugging some types of problems at [WWW] http://wiki.ubuntu.com/DebuggingProcedures

At a minimum, we need:
1. What email providers you use?
2. specifically, with which email providers do you have this issue?
3. Do you have filters on? Any filtering the inbox?
4. Do you have a search active on the inbox (look at the search field above the email headers)?

5. Finally -- and perhaps quite important --: if you reinstall 2.22.2, do you still see this behaviour?

Please keep in mind that Hardy-proposed is still a test environment -- we believe the package is sane, but still wish for more people to test it. You might indeed have found a bad bug.

Thank you.

[C de-Avillez (hggdh2)]

huh, sorry, by 'email providers' I mean protocols. GMAIL IMAP would be a protocol by itself.

[mstuermer (matthias-stuermer)]

1. I only use POP, but from 3 different providers including GMAIL. When looking at the around 10 out of 50 emails which were not displayed by Evolution but are shown by Thunderbird (both use mailbox files), the 10 'missing' mails are from all different providers. Therefore I don't see a pattern in this respect.
2. stuermer.ch; ethz.ch; gmail.com
3. Yes I use many filters, but the 10 missing mails should not have been catched. Also when searching for their titles in all accounts there are no results. So the emails are really no visible in Evolution.
4. No
5. After a lot of hassle with Synaptic I managed to reinstall 2.22.2. Unfortunately I still have this behaviour, I don't see the 'missing' mails with Evolution but I see them in Thunderbird (even after deleting the index files in the .evolution folder)

One more thing I can provide: The earliest 'missing' email was received on 2008-07-08 17:51 CET. If you wish I can send you the 'missing' mails in a private message so you can analyze the headers. I don't see a pattern, but I'm not very familiar with email headers.

Thanks for your help!

[C de-Avillez (hggdh2)]

this is weird... lets try one bypass before we actually look into it: on a terminal:

0. quit Evolution
1. tar -cvf savedmail.tar ~/.evolution/mail/local
2. cd ~/.evolution/mail/local
3. rm *.ev-summary
4. restart Evolution -- check if *all* emails are now shown

This will tell us if the summaries got corrupted (I do not think *all* of them would, but, just in case...

[mstuermer (matthias-stuermer)]

OK, I did what you said for the ~/.evolution/mail/local directory but the missing mails didn't appear.

It's extremly strange, I know. Here I can show you exactly what I mean:

I went into my Ubuntu folder within Evolution and took a screenshot of all the mails concerning this bug report. As you can see in the attached file only four emails are listed. However, when looking at the Ubuntu mailbox file and getting out the last 400 lines there are 6 emails - your first two mails are missing, but not the newest one! (this also shows that the filters still work correctly - at least in this case)

So you really have to trust me, I'm not fooling around, I'd love to use my time for more productive work... I use Evolution since the early Ubuntu days (more than 3 years ago) but have never seen such a thing.

Let's hope you can find this nasty bug - or give me a hint what I did wrong!

[mstuermer (matthias-stuermer)]

That's the screenshot in Thunderbird of the mailbox file - the two missing messages are shown.

[C de-Avillez (hggdh2)]

mstuermer, I do not doubt you... Let's try another track: try Edit/Show Hidden messages, and let's hope it is a simple thing.

Otherwise, we will need to start Evo on a trace: stop Evo and then, on a terminal, 'env CAMEL_DEBUG=all evolution --component=mail > evo.log 2>&1', and lets wait until you find yourself with a NEW missing message. Then, if you want, you can email me the log directly (hggdh2 at gmail dot com). And cross your fingers ;-)

[mstuermer (matthias-stuermer)]

Sorry for not replying earlier. I tested "View | Show Hidden Messages" without success. Then trying to log with "env" resulted in a 60MB log file which I don't think will help you any further.

One thing I'd like to know from you: If you take the attached file "DemoEvolutionError_MailboxFile", put it into your mail folders and look at the folder from within Evolution, what do you see? As you can see in a text editor, the Mailbox file contains *six* email messages. However, in my Evolution only *four* messages are displayed (see also my screenshot in a message above: http://launchpadlibrarian.net/15941361/ScreenshotAndMailboxFile.zip ). So how many do you see in your Evolution?

[This Born (this)]

I did experience the same problem, when I copied the mailbox file mentioned above to ~/.evolution/mail/local/Inbox.sbd. Two of the six e-mails were missing.

[C de-Avillez (hggdh2)]

I can confirm it also, and I am running trunk (2.23.5)... now to find out what gives here.

[Pedro Oliveira (pmsfo)]

I have the same problem since 4 day ago, the messages are downloaded but don't not apear. On 3 diferent accounts, gmail, hotmail with inetd, and imap.

[C de-Avillez (hggdh2)]

mstuermer, sorry for the delay.

I downloaded your mailbox file, and had a look at the headers, and I cannot see any missing critical header (which would be a good reason for a non-displayable email).

I then added the mailboxfile into my Evo (2.23.4 and 2.23.6 trunk). *Both* of them only show 4 emails. Of course -- and this is already clear for you -- the mailboxfile actually contains more than that.

Making a long story short, I had to move into KDE (from Gnome) due to X suddenly eating up all my CPUs (*seems* to be related to using the Radeon video driver *plus* GTK applications). While at KDE I set up Kmail, and imported all my local mailboxes (most of my email is under IMAP, with just a few local folders). I then looked at your mailboxfile -- and found I could see all emails.

I am now back to Gnome, and Evolution (but out of the Radeon driver :-), now running trunk (2.23.6+), and I still cannot see all emails under Evo.

So... I think this warrants an upstream. I am setting it to triaged, and will open an upstream & attach your sample mailboxfile.

Back in a few.

[C de-Avillez (hggdh2)]

marking as triaged, and setting affects for http://bugzilla.gnome.org/show_bug.cgi?id=546438

[vilbara (vilmantas-baranauskas)]

I can confirm this behavior with MS Exchange server accessed over IMAP. New messages are not shown in Evolution's INBOX.

Closing evolution, removing ".evolution/mail/imap/..." directory and starting evolution helps - it works then for some time, e.g. half a day and then stops displaying new messages again.

This started when I updated to Ubuntu 8.10 beta.

[vilbara (vilmantas-baranauskas)]

More info.

If I right-click on "Inbox" and select "Properties", then the dialog displays the right number of messages in the inbox.

Screenshot shows this weird situation.

[dnh (dnh)]

I can confirm this bug, I have had it for a while using the Exchange-plugin. New mails do not show. In addition, running a fresh install of 8.10, now the contents of some mails are not shown.

[w.chris (b-admin-witek-fr)]

I confirm this bug as well. This is an urgent thing. I use Evolution in a professional environment. A rapid correction would be really appreciated.
The missing emails appear if I remove all "courierimapkeywords/ courierimapsubscribed courierimapuiddb" files directly on the imap server. But this isn't a good solution because the new emails in those folders can still not be displayed for some reasons.
I noticed this problem not only in the inbox but in the outbox also.

This is very urgent, please help !
Thank you.

[w.chris (w-chris)]

I could reproduce the bug. These are the steps :

- I create a new folder in 'sent-mail' called 'test' (the name doesn't matter),
- I send a new email. The copy of the new email appears automatically in the 'sent-mail' folder,
- I move the new sent email to the previously created folder 'test',
- I restart Evolution,
- I open the email in the 'test' folder,
- I come back to the 'sent-mail' and it appears empty.

As you can see on the screenshot, there are 35 messages in 'sent-mail'. There are not visible and all operations (copy & move) to this folder are not possible. I cannot select this folder if I want to copy emails into it for example.

This bug is very serious. Why the 'importance' of this post is set to 'Med' ???
It should be 'Hi'. This bug makes Evolution not usable.

Quick help needed.

[C de-Avillez (hggdh2)]

@all: please select view hidden messages (View/Show Hidden Messages), then verify if the messages appear; Also please check the Junk folder.

[w.chris (b-admin-witek-fr)]

it has nothing to do with hidden messages. The folder in which
messages are not displayed is not accessible. As I said I cannot
access it anymore with evolution if I want to copy or move messages in
it. I cannot select that folder while moving or copying messages. It's
not selectable. Clicking on 'view hidden messages' doesn't help at all.

--
K. Witek
Administrateur Réseaux & Systèmes
http://www.witek.fr

hggdh <email address hidden> a écrit :

> @all: please select view hidden messages (View/Show Hidden Messages),
> then verify if the messages appear; Also please check the Junk folder.
>
> --
> Some emails are not displayed but are saved in the mailbox file
> https://bugs.launchpad.net/bugs/247364
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in The Evolution Mail & Calendaring Tool: New
> Status in “evolution” source package in Ubuntu: Triaged
>
> Bug description:
> Binary package hint: evolution
>
> Since yesterday's update (2008-07-09) a few but not all new emails
> are not displayed in Evolution. When I realized there were missing
> emails I looked into the mailbox file of my Inbox ("tail -n 1000
> .evolution/mail/local/Inbox") and found that the emails were
> downloaded but not displayed! That's really very bad and made me
> switch to Thunderbird after over 3 years of more or less happy using
> Evolution.
>
> evolution:
> Installiert:2.22.3.1-0ubuntu1
> Mögliche Pakete:2.22.3.1-0ubuntu1
> Versions-Tabelle:
> *** 2.22.3.1-0ubuntu1 0
> 500 http://ch.archive.ubuntu.com hardy-proposed/main Packages
> 100 /var/lib/dpkg/status
> 2.22.2-0ubuntu2 0
> 500 http://ch.archive.ubuntu.com hardy-updates/main Packages
> 2.22.2-0ubuntu1.2 0
> 500 http://security.ubuntu.com hardy-security/main Packages
> 2.22.1-0ubuntu3 0
> 500 http://ch.archive.ubuntu.com hardy/main Packages
>

----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.

[C de-Avillez (hggdh2)]

@w.chris: please note: (1) the original reporter had an issue with the Junk plugin; (2) you are not the only one complaining about it. So the questions still stand. For you, it might be interesting to verify what IMAP folders you are subscribed to.

@all: please specify your Evo version, and what type of mail this happens on (local mbox, IMAP, Exchange). I would like a debug run to be done:

1. close Evo;
2. open a terminal and:
2.1. evolution --force-shutdown
2.2. CAMEL_DEBUG=all evolution > evo.log 2>&1
3. on the newly opened Evo, reproduce the issue;
4. close Evo;
5. verify the 'evo.log' file not to contain private or sensitive data;
6. upload it here.

And we will go from there.

[w.chris (b-admin-witek-fr)]

there is indeed a second problem with the Junk plugin. I've just experienced it before I read your message ! The junk filter moved 1300 messages from the inbox to the Junk folder !!! What is going on with Evolution ????
But unfortunately it's not the only problem and has nothing to do with what I described in my previous post.
 Can we focus on the first problem ?

Evo version: 2.22.3.1 on Hardy with one imap account.

You cannot really see the issue with CAMEL_DEBUG=all. I attached it anyway. Lines are printed out when I browse all other folders but nothing is printed out when I click on the 'sent-mail' (the folder in with the issue). I curiously I have the 'noselect=yes' line only for the 'sent-mail' folder:
uri: imap://<email address hidden>/INBOX/sent-mail;noselect=yes

[C de-Avillez (hggdh2)]

@w.chris: First to your issue with not seeing the IMAP folder: please check your IMAP subscriptions (Folder/Subscriptions; then select your IMAP server, and verify all needed folders are checked -- i.e., subscribed to).

On the Junk plugin: I do not know why this happened, but it is a well-known issue you must train your Junk plugin (or disable it). See http://www.go-evolution.org/FAQ#Spam.2FJunk.

[C de-Avillez (hggdh2)]

@w.chris: additionally, your first comment here states:

"The missing emails appear if I remove all "courierimapkeywords/ courierimapsubscribed courierimapuiddb" files directly on the imap server."

This suggests something your IMAP server (which, I guess, is the courier implementation) is doing, not really to Evo. And, again, suggests IMAP folder subscription as the cause.

[w.chris (b-admin-witek-fr)]

first of all: I could see all of my email folders, they were all checked in IMAP subscription. I think I didn't make myself clear. I could see the sent-mail folder but it was empty and I couldn't copy or move messages in it.

I was expecting that somebody would tell me that the problem was with the imap server. To my mind, I don't think so. The courierimap was installed with a default Maildir configuration and about 20 people have been using it for 4 years with Kmail, Outlook and Horde(webmail). Problems appeared when everybody migrated from Kmail to Evolution (we had some different issues with kmail, nothing to do with this problem).
But I don't have time to discus about this or prove that the problem is not related to the imap server because my goal is to get rid of this problem. So I decided to migrate from courier to dovecote. I did it yesterday and it seems that the problem is solved. All folders can be accessed and all messages are visible.
Again, my goal was to get rid of this problem and it seems that this problem is solved for me. But I still think that this is a bug in Evolution and not using courierimap is not a solution. Anyway I won't be able to furnish more info about the problem because I don't use courierimap any more.
 Thank you for your help, I really appreciated it.

[C de-Avillez (hggdh2)]

Well, I did *not* say your problem was with the IMAP server. I asked you to check on the subscriptions. I use IMAP also; I never used Courier-IMAP, but always Dovecot.

It is quite possible that Evo is doing something not quite right, or (more probably) not quite the Courier way; also there is IMAP and IMAP4rev1 -- and it seems Evo has some issues with IMAP4rev1 (and I never used IMAP4rev1). So any of these may have been affecting you.

Anyway, as you say, you just wanted to get rid of your problem, not to solve it. So there.

[WolphFang (mjoyner-vbservices)]

I am experiencing this problem with Exchange server 2003, this is not an IMAP specific issue.

[ggiese (ggiese)]

Same here with evolution-exchange under Evolution 2.24.3.
I can however make the missing files visible by doing the following:
- use outlook (I use 2k7) under a windows machine to copy all data from the inbox to a different folder on the exchange server.
- wait for evolution to show an empty inbox.
- move the files back the same way.

This seems to clear something on evolution that makes it possible to see the mails again.

[robanon (robert-mcdonald)]

I'm also experiencing this problem with Exchange server 2003, Evolution 2.24.3 using the Exchange Plugin.

To make a missing email display, I can go into Outlook Web Access, move the affected email to another folder, then move it back again.

Clicking Send/Receive in Evolution then displays the missing email.

Right clicking on Inbox and selecting properties in Evolution shows the correct number of emails on the Exchange server, which is less than the amount shown in Evolution's inbox.

Selecting view hidden messages (View/Show Hidden Messages) does not show the missing messages.

It seems that the missing emails do eventually show up, but up to a few days later, I'll see if I can get more definite information regarding that though.

I'm surprised the importance of this bug has only been classified as medium, I would have thought losing emails in your inbox would classify as pretty urgent?

[robanon (robert-mcdonald)]

Just some more info on my Evolution setup:

"Check new messages for Junk contents" has never been enabled and is still disabled.
"Apply filters to new messages in Inbox" was enabled, I've now disabled it.
"Synchronise account locally" was enabled, I've now disabled it.

[Sebastien Bacher (seb128)]

the current comments seem to be exchange issues

[robanon (robert-mcdonald)]

The Exchange server that I am connecting is one of several Exchange servers used by several hundred people. These Exchange servers have been working for years.

There are a few people that have started using Evolution to connect to the Exchange servers and they are the only people having this problem.

You can see the missing emails in OWA and Outlook.

It must be a problem with Evolution.

[Sebastien Bacher (seb128)]

nobody denied that's a client issue but that would be evolution-exchange if that was exchange specific

[robanon (robert-mcdonald)]

I've added an IMAP account to check the same Exchange account for emails to verify if it is an evolution-exchange specific problem.

The thing is the problem is very intermittent, I'd guess its around 1 in a hundred emails don't display on time, and if the email shows up a few hours or even a day late you might not realize there had been a problem.

I'm also not sure if emails occasionally don't show up at all, because its difficult to know about emails you didn't receive.

But if the problem is exchange specific, does that rule out trying to fix it? Or does it make the the problem less urgent?

The main advantage of Evolution to me (and I guess many others) is that it can integrate with Exchange servers, as far as I know its the only Open Source MUA that works with Exchange's calendaring etc, but it requires the evolution-exchange plugin to work.

If I had an alternative MUA I'd move to it because of this problem, but need Evolution for the calendar support.

Hopefully we'll get to the bottom of it soon, I'll let you know how I get on.

[Sebastien Bacher (seb128)]

> But if the problem is exchange specific, does that rule out trying to fix it? Or does it make the the problem less urgent?

no, but it means it should be assign to a different product where people having access to exchange server can work on it, I've no exchange server and can't work on those issues, opening the bug on bugzilla.gnome.org where the people writting the code will read it on the evolution exchange component would be a good idea too

[François Tissandier (baloo)]

My 2 cents on this pb:

-I noticed it maybe one or two weeks ago, but of course I can't be sure it did not happened before
-was on Intrepid, I'm on Jaunty now
-connected by IMAP to an Alt-N Mdaemon server

-wrong number of emails, only 180 when in fact I had 1800 (total confirmed simply by displaying the properties of the Inbox...)

-I tried to delete my evolution configuration files, reconfigure the imap account, same problem

-I tried to unsubscribe from the Inbox, then subscribe again, sometimes it was giving me the emails back, but only during a few seconds. Sometimes no difference. What I noticed was that they were disappearing AFTER a filtering (you can see it in the status bar at the bottom of Evolution when it's filtering)

-I didn't configure the spam filter for this IMAP account, but I had some filters, so I deactivated the filters, no change. However, each time I tried to unsubscribe /subscribe, it was STILL filtering. I didn't really understand why, no spam filter, no mail filters set for this imap account.

-then I discovered this: in the general options (NOT the account specific ones), the spam filter was activated! I deactivated it, and since, I don't see any filtering anymore when I unsubscribe/subscribe or just receive emails. And I see ALL my emails... minus 1! 1885 displayed, properties window says 1886... But that's a big improvement to me already. And it makes Evolution faster, as it's not filtering for spams anymore.

Can someone check her/his own settings and see if it's related to this spam setting?

[rodelasangelo (angelo-rodelas)]

good morning.

We have the same problem in the company. Some of our Ubuntu - Evolution users are experiencing bug, which is really bugging our department. Complaints are coming in and out.

temporarily, I have done a TEDIOUS solution. I copied the Inbox (mbox file) located at home/evolution/email/local and converted it to eml. then exported it to MS OUTLOOK (XP Computer). All the emails show there, from the oldest to the latest. Like i said, this is tedious, so i hope this really big-bugging-bug be solved, immediately.

if you have an easier way, where i dont have to copy-convert-import emails, please to email me!

[kilmarnock (kilmarnock-gmx)]

good morning.

same problem here. I use cyrrus imap server. thunderbird sees all messages. squirrel mail sees them. evo not. evo is 2.26.1, ubuntu 9.04.
I tried the tricks above without success.

[Sebastien Bacher (seb128)]

you can also comment on the upstream bug

[nickleus (nick-humphrey)]

i also was experiencing mails not coming into my inbox, sometimes they would, sometimes they wouldn't, it was weird. then i checked my junk folders. sure enough there were the missing emails. in my plugins list there are 2 junk/spam plugins: bogofilter and spamassassin
anybody have any idea which one (unless both) is the culprit?

[nickleus (nick-humphrey)]

btw, i'm using evo 2.26.1 on ubuntu 9.04

[Jessica (jessica-heinrichs)]

hi,

i had the same problem, the junk filter moved all incoming mails (my normal emails, too) into the junk-folder.
than i disabled the bogofilter and installed spamassassin (in synaptic) instead and chose this one in Edit-->Preferences-->Mail-Preferences-->Junk . until now it is working. the spamassassin-filter and my normal filters work, too.
so i think it's a problem in the bogofilter.
hope it helps.

the picture shows my preferences (sorry that it's german)

[kilmarnock (kilmarnock-gmx)]

Hi!
The bugs described by nickleus and Jessica are different issues. The difference is simple:

The mails _are_ in the inbox folder. Nothing moved them nowhere. Thunderbird f.e. can see them there.

cheers
Felix

[RaveDigger (ravedigger)]

I had a similar issue, not sure if this is relevant, but I'd like to contribute in case it helps someone. For some reason, not all of my messages would show when I had my view set a certain way. When using the vertical preview, right clicking and choosing sort by received is not the same as right clicking and sorting by date.

[Matej Kenda (matejken)]

This is happening on Karmic as well using Exchange server.

Reproduction instructions (not 100% reliable):

1. read e-mails via Outlook Web Access: open new mails
2. close OWA and open Evolution
3. Some of the e-mails read with OWA will not show up, but the count on the folder will indicate unread e-mails

Clearing local cache solves the situation.

[Matej Kenda (matejken)]

My colleague pointed me to a different bug in gnome, describing the same issue:

https://bugzilla.gnome.org/show_bug.cgi?id=581908

It is marked as resolved. Did it make it into Ubuntu version of Evolution as well?

[JeffV (jeff-launchpad-tanasity)]

Aha! I have had this bug too but...

I have just found that Evo is independently classifying some messages as junk and storing them in the Junk folder. In Thuinderbird they still show up in the inbox but in Evo they don't appear in the Inbox!

This may be because of the Spamassassin plugin.

[PrototypeX29A (preineke)]

I've had a similar problem, where I wasn't able to see all may incoming mails.
In turned out that there was some persistent mail filter, which was not clearly visible.

"Search > Clear" removed the filter.

[Matej Kenda (matejken)]

I switched from using Exchange connector to IMAP/LDAP combination to access the same mail server (Exchange) and I am able to read e-mails without any problems. I assume that in my case the problem was related to Evolution Exchange connector, which I believe is unreliable.

[Matej Kenda (matejken)]

Soon after writing the previous comment the problem appeared again using IMAP access to Exchange Server.

Reproduction instructions:

1. Read e-mails in Evolution (Inbox)
2. Close Evolution
3. Wait for some new e-mails to arrive and read them in Outlook Web Access
4. Close OWA and start Evolution: Voila! Mail missing in local cache

[Matej Kenda (matejken)]

I removed the file folders.db for this particular IMAP server and started Evolution.

The file was created from scratch on next start and the mails re-appeared.

[Jörg Frings-Fürst (jff-de)]

hardy not longer supported
please change the status to Won't fix

[Jörg Frings-Fürst (jff-de)]

change status to invalid