KMail duplicates emails

Bug #883459 reported by Vladimir Kolesnikov
48
This bug affects 10 people
Affects Status Importance Assigned to Milestone
KDE PIM
Confirmed
Medium
kdepim (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

KMail duplicates emails from time to time.

Please see the screen shot attached.

This happens every day, several times a day and is really annoying.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: kmail 4:4.7.2+git111007-0ubuntu1
ProcVersionSignature: Ubuntu 3.0.0-12.20-server 3.0.4
Uname: Linux 3.0.0-12-server x86_64
NonfreeKernelModules: fglrx
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
Date: Sat Oct 29 15:23:06 2011
ExecutablePath: /usr/bin/kmail
ProcEnviron:
 LANGUAGE=ru_RU:ru
 LANG=ru_RU.UTF-8
 SHELL=/bin/bash
SourcePackage: kdepim
UpgradeStatus: Upgraded to oneiric on 2011-08-21 (68 days ago)

Revision history for this message
In , Maednoldor (maednoldor) wrote :

Version: 4.7 (using Devel)
OS: Linux

I have a Jira account. I have set up 2 filters: one is used to move messages for JIRA folder and mark it as read. Filtered by sender (my own activieties).
The second one is used to move all other JIRA maessages to the same folder, but do not mark them as read.

As a result, I have a lot of duplicated messages in my JIRA folder. Some messages (during 2-3 weeks) became duplicated to 10 times.

Reproducible: Sometimes

Steps to Reproduce:
1. create some filters, as I mentioned.
2. try to get the mail.

Actual Results:
New messages arrive in the INBOX folder (that are already placed to the JIRA folder). After this, KMail starts to filter them and send the duplicates to JIRA folder again.

Expected Results:
Messages should be placed to the folder. No duplicating.

As I remember, one of e-mail clients, used by mine, has a function 'remove duplicates'. This should be very useful in this case.

11 comments hidden view all 252 comments
Revision history for this message
Vladimir Kolesnikov (sjinks) wrote :
Revision history for this message
Alessandro Menti (elgaton) wrote :

Thanks for reporting the bug.

Could you please specify in more detail when the duplicates start appearing? For example, are they already there when you start the program, after you move/copy some messages... or does this bug appear randomly? If possible, could you provide a test case (a sequence of actions that can be performed to reproduce the bug)?

Changed in kdepim (Ubuntu):
status: New → Incomplete
Revision history for this message
Vladimir Kolesnikov (sjinks) wrote :

From what I can tell, this has to do something with filters - as it happens only on accounts I have set up filters for (e.g., if "Anywhere in Header" contains "X-Launchpad-Hash", move email into "KMail Folders/Launchpad").

Emails get duplicated usually when new mail arrives.

Sometimes emails resurrect after I delete them.

This makes me think that when email is moved by the filter rule, KMail does not actually delete it - just hides it. Then when new mail arrives, it finds that "semi-deleted" email, applies a filter to it and now I get a duplicate of the existing emails.

I often see a warning "KMail Folders: Cannot modify emails in folder" (the warning is in Russian so my translation is approximate).

Revision history for this message
Vladimir Kolesnikov (sjinks) wrote :

Another example: I received these emails: https://url.odesk.com/u2fi4 (screenshot)

I read them and deleted them.

Then on the next mail check I see this: https://url.odesk.com/7t69v (screenshot)

That new email is the one I have deleted earlier.

The problem is that it does not happen everytime with every email - but often enough to be very annoying.

Revision history for this message
Alessandro Menti (elgaton) wrote :

(Reassigned the bug to the right package.)

Are you perhaps using an IMAP server? I have run a search on the KDE bugtracker and found the following bug:
<https://bugs.kde.org/show_bug.cgi?id=95064> - Duplicates messages in imap folder after filtering
It is related to an old version of KMail, but it seems (at least from the last comments) that it occurs on some newer versions as well.
If so, could you please tell me so I can link this bug report to the one on the KDE bugtracker?

Changed in kdepim (Ubuntu):
status: Incomplete → New
Revision history for this message
Vladimir Kolesnikov (sjinks) wrote :

The problem occurs only on GMail's POP3 account (actually that's the only account that uses filters).

Two more observations:
* only filtered emails get duplicated/resurrected (e.g., only when emails were moved from one folder to another);
* this only happens when the filter has to process several emails at once (I do not remember seeing this behavior when only one email comes - only when there are multiple emails).

Revision history for this message
Vladimir Kolesnikov (sjinks) wrote :

One more example:

1. 10 emails have come (screenshot: https://url.odesk.com/kze6s ). i have read and deleted them.
2. 30 minutes later (or so) when new emails came: https://url.odesk.com/njyqc

monit alerts (both), lfd on snowboarding-mail-0 and abuse report (basically 4 of 5 emails KMail notified me about) are "resurrected" emails - they appeared after I deleted them.

Revision history for this message
Alessandro Menti (elgaton) wrote :

OK - I think that this other bug report (<https://bugs.kde.org/show_bug.cgi?id=283682> - KMail duplicates filtered messages) matches both the problem and the version of KMail you are using. (It may be a bit difficult to reproduce the bug since it partially happens at random - I'm referring to the deletion part).

If you find that the bug report matches your problem too (I'm not 100% sure), could you please leave a comment in the KDE bugtracker (providing a link to this bug report); otherwise, could you please open a new bug report at the KDE bugtracker (<https://bugs.kde.org/>), filing it against the product "kmail2" and the version "4.7"? This way, the KMail developers will be notified of the bug and will fix it.

In any case, after you have completed the process, please click on the link "Also affects project" at the top of this bug report and write the URL of the bug in the KDE bugtracker there. This way, we can notice when the bug has been fixed by the KMail developers and release a new version of KMail in Ubuntu accordingly.

Thanks again for all the information!

5 comments hidden view all 252 comments
Revision history for this message
In , Vladimir Kolesnikov (sjinks) wrote :
4 comments hidden view all 252 comments
Revision history for this message
Vladimir Kolesnikov (sjinks) wrote :

Done.

Revision history for this message
Alessandro Menti (elgaton) wrote :

Thanks again!

Changed in kdepim:
importance: Unknown → Medium
status: Unknown → New
4 comments hidden view all 252 comments
Revision history for this message
In , Montel-3 (montel-3) wrote :

*** Bug 286053 has been marked as a duplicate of this bug. ***

Revision history for this message
In , Geoff Clements (geoff-v8x) wrote :

I have the same problem, nearly all messages in my inbox are duplicated. I use only IMAP so don't know if this affects pop3 accounts. I do have filtering enabled.

Revision history for this message
In , Rick Vernam (rickv-hobi) wrote :

Running Gentoo stable, just pulled in kmail2. my pop account is showing multiple copies of messages that have been moved to another folder by a filter rule. Also, when I click on the Inbox messages that had been moved by filter suddenly appear in the inbox, then are moved back into the the folder specified by the filter rule (and thus the duplicate).
Also checking email will cause the filtered messages to duplicate.
This is not happening with my IMAP accounts (though none of my filters affect messages that typically come in through the accounts I access via IMAP).

Happy hunting.

Revision history for this message
In , Rick Vernam (rickv-hobi) wrote :

I exported all my filters, and deleted them from KMail2. KMail2 is now running without filters. If I move an email from the inbox to another folder (ie, the "destination" folder), the email will appear again in the inbox (and remain in the destination folder as well). This occurs both when the destination folder is a sub-folder of the inbox, and when the destination folder is directly under the Local Folders.

Revision history for this message
In , Pragma-n (pragma-n) wrote :

I experience this problem in 4.7.90 (4.8 beta2).

If a mail just downloaded is moved by a filter to a different folder then there is a good chance that it will re-appear again in the original folder at the next update

It does not happen with all the accounts or all the filtered messages. However if a message is affected then it's 100% reproduceable: it will be duplicated at every folder update (forever?).

Changed in kdepim:
status: New → Confirmed
Revision history for this message
In , Ggrau (ggrau) wrote :

Same here... with 4.7.4 on Fedora16
Most mails I try to move from my Inbox will appear again as duplicate.
This happens also, when moved by filter rule.
Pressing <Del> to move them into trash surprisingly seems to work.

Revision history for this message
In , Matthias-nagel-3 (matthias-nagel-3) wrote :

Same problem for me. I am using kmail 4.7.4-r1 from the Gentoo Stable amd64 repository. I have to POP3 accounts and some filters that examine the mail header (mostly the "from" attribute) and move the mails into folders and marks them as read. Sometimes new messages are duplicated. This also affects mails that are passed through the filter but not modified, because the filter condition does not apply.

The mails are not duplicated immediately but reemerge after a folder update, especially after starting kmail. Suddenly mails that are already some days old and have already been processed by the filter show up again.

I can observe the same behaviour as in comment #6. Not every mail that is processed by some filter is affected, but if some mail is affected once, the mail reappears over and over again. I have the feeling that a mail is more likely to be affected if the filter has to process a lot of new mails at once.

Revision history for this message
In , Tokoe (tokoe) wrote :

Hej all,

can you please check whether akonadi reports some errors regarding the target folders you try to move the mails to, about a missing remote id?
You can also use akonadiconsole and check the folder properties on the 'internal' tab to find out the remote id.

Changed in kdepim:
status: Confirmed → Incomplete
Revision history for this message
In , Matthias Mailänder (mailaender) wrote :

Found this while trying to reproduce and understand akonadiconsole logs:

AgentBase(akonadi_imap_resource_0): Select failed, server replied: A000068 NO Mailbox doesn't exist: &APw-lleimer

The local Trash folder is called "Lokale Ordner/Mülleimer". This is where the spam check wants to move spam. Putting it on the IMAP Junk folder as Thunderbird does is not possible/allowed.

Revision history for this message
In , Geoff Clements (geoff-v8x) wrote :

(In reply to comment #9)
> Hej all,
>
> can you please check whether akonadi reports some errors regarding the target
> folders you try to move the mails to, about a missing remote id?
> You can also use akonadiconsole and check the folder properties on the
> 'internal' tab to find out the remote id.

Tobias,

Sorry this will sound lame but I'm not sure where to look for the akonadi error log. I did try this:

$ grep -R remote ~/.local/share/akonadi/
grep: /home/geoff/.local/share/akonadi/socket-dhcppc2: No such file or directory
grep: /home/geoff/.local/share/akonadi/socket-dhcppc1: No such file or directory
Binary file /home/geoff/.local/share/akonadi/db_data/ibdata1 matches
Binary file /home/geoff/.local/share/akonadi/db_data/akonadi/pimitemtable.frm matches
Binary file /home/geoff/.local/share/akonadi/db_data/akonadi/collectiontable.frm matches
Binary file /home/geoff/.local/share/akonadi/db_data/ib_logfile0 matches
grep: /home/geoff/.local/share/akonadi/akonadiserver.socket: No such device or address
grep: /home/geoff/.local/share/akonadi/socket-scotty/akonadiserver.socket: No such device or address
grep: /home/geoff/.local/share/akonadi/socket-scotty/mysql.socket: No such device or address

As the only matches are in binary files I didn't take this any further.

Using akonadiconsole there is nothing at all displayed in the browser->internals tab. I do notice that non of the remote IMAP folders are displayed in the panel on the left hand side. I see only:

Personal Contacts
Local Folders
    |
     - Outbox
Search

Revision history for this message
In , Ggrau (ggrau) wrote :

All I can see when moving a Mail is:

==================================
NotificationManager::notify ( Item (129274, 1329071216.R951.neptun:2,S) from collection 467 movedto collection 535 )
==================================

When rescanning the derectory, I just moved it out with F5:
========================================
NotificationManager::notify ( Item (129275, 1329071216.R951.neptun:2,S) in collection 467 added )
NotificationManager::notify ( Item (129275, 1329071216.R951.neptun:2,S) in collection 467 modified parts (PLD:RFC822) )
=======================================

There is a lot of debug traffic in the upper window, but (as I can see) nothin g suspecious.

Revision history for this message
In , Milian Wolff (milianw) wrote :

Tobias, please give us more information on what exactly we should look at and post here.

Revision history for this message
In , Milian Wolff (milianw) wrote :

rightclick on (some) of my target folders in akonadiconsole shows that they all have a "remote identifier" set.

I note though, that the situation has improved, I don't get as many duplicated emails anymore as I used to before. I'll see whehter I can reproduce it at all, maybe this is actually fixed in recent git?

Revision history for this message
In , Christian Trippe (christiandehne) wrote :

*** Bug 282296 has been marked as a duplicate of this bug. ***

Revision history for this message
In , Geoff Clements (geoff-v8x) wrote :

(In reply to comment #9)
> Hej all,
>
> can you please check whether akonadi reports some errors regarding the target
> folders you try to move the mails to, about a missing remote id?
> You can also use akonadiconsole and check the folder properties on the
> 'internal' tab to find out the remote id.

Tobias,

I have checked a number of times now in akonadiconsole, the debugger does not show any error messages when the messages are duplicated. Please note however that the duplicated messages are the ones that are /not/ moved because they do not meet any filter criteria. The messages that are moved by the filters are not duplicated.

Revision history for this message
In , Matthias-nagel-3 (matthias-nagel-3) wrote :

Opposed to comment #16 the mails that are moved are duplicated with me.

Even mails that are not moved by a filter but that are move manually (drag and drop) are duplicated. Mails that are "moved" to the bin folder by hitting "DEL" are duplicated, too. In all three cases the mail re-appears in the original folder after some period of time. The problem can be avoid by copying the mail to the destination folder and then purge the original mail from the source folder by hitting "Shift+DEL". This work-around also applies to the filter. If I create two filter actions - copy mail first, then purge the original - everything is fine.

And in all three cases there is no error report in the akonadiconsole.

Revision history for this message
In , Milian Wolff (milianw) wrote :

reopening, as I also just noticed this again. The target folder has a proper remote identifier set, yet I saw that some mails where duplicated by the filter manager...

Tobias, any other ideas what I could look into?

Changed in kdepim:
status: Incomplete → Confirmed
19 comments hidden view all 252 comments
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in kdepim (Ubuntu):
status: New → Confirmed
Revision history for this message
Alexandre (ab-linuxfr) wrote :

I have the same problem, but with 4.8.1.

19 comments hidden view all 252 comments
Revision history for this message
In , Alexandre-bonneau (alexandre-bonneau) wrote :

This nasty bugs affect 4.8.1 too.

Revision history for this message
In , Dr-bugzilla (dr-bugzilla) wrote :

I believe the problem I have in 4.8.4 is the same problem: emails passing through filters are sometimes being duplicated.

I have a cron job on my servers which send me an "hourly test email" message which I receive on my desktop PC; these have been running for about two years without trouble. I have upgraded my desktop from Ubuntu 10.04 and kmail to Xubuntu 64-bit 11.10 and kmail v4.7.3 on my desktop and I am now seeing some of these hourly emails as identical duplicates (same ids in the headers, etc.) in their folders in kmail2.

For example, my PC was unattended for 11 hours last night and during that time 3 of the hourly emails appear as duplicates in their folder.

The emails are fetched by kmail2 using POP3 and filed into mbox folders by filters. When using kmail I had it set up to remove emails from servers; I set kmail2 to leave it on the servers (just in case I reverted back). On the server I can see there are no duplicates so they were not sent as duplicates.

The rule checks the subject of the email contains "Hourly Test Email" and the "Envelope-to" equals the email address I specified for this sending server. it then moves the email to the correct folder and processing rules is then stopped.

These email checking filters are applied before "Bogofilter check", "Spam Handling" "Classify as Spam" and "Classify as NOT spam" filters, so they are all probably innocent.

The email was downloaded and processed by "interval email checking".

To reproduce it:
1. Set up a filter to receiving incoming email and automatically file it in a folder.
2. Send lots of emails to it.
3. Check the folder.

I just did this by manually sending 18 emails; one of them in the middle was duplicated.

I set up a script to send an email every 5 seconds and sent 77 emails. 79 arrived; two had been duplicated.

The package I have installed is: kmail 4:4.7.4+git111222-0ubuntu0.1
"About KMail" says "KMail Version 4.7.3, Using KDE Development Platform 4.7.4 (4.7.4)"

Revision history for this message
In , Alexandre-bonneau (alexandre-bonneau) wrote :

Just for information, the bug is still seen in 4.8.2.

Revision history for this message
In , PhobosK (phobosk) wrote :

Affects heavily KDE/KMail 4.8.4 too...
The problem is definitely in moving messages by an applied filter. Manual filtering doesn't have it.
The affected messages are marked for deletion on the remote server (IMAP in my case), deleted from the KMail folder and then reappearing back in the same Kmail folder and in the remote folder too. So at some point in time, the remote folder has the marked for deletion messages + one fresh new and duplicate of all marked for deletion. The strange thing is that some of the duplicates have only headers and not the full messages.
So the problem seems to be in the Move action function of the filtering rules.

Revision history for this message
In , Andras Mantia (amantia) wrote :
Download full text (5.0 KiB)

Git commit d64d61b470c56b92294f6adee6d74305a217628d by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'master'.

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364

M +7 -11 kmail/kmcommands.cpp
M +10 -7 kmail/kmcommands.h
M +1 -1 kmail/kmmainwidget.cpp
M +1 -5 mailcommon/filter/filteraction.cpp
M +6 -5 mailcommon/filter/filteraction.h
M +6 -0 mailcommon/filter/filteractionaddheader.cpp
M +2 -0 mailcommon/filter/filteractionaddheader.h
M +2 -2 mailcommon/filter/filteractionaddtag.cpp
M +1 -1 mailcommon/filter/filteractionaddtag.h
M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp
M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h
M +7 -0 mailcommon/filter/filteractionbeep.cpp
M +1 -0 mailcommon/filter/filteractionbeep.h
M +18 -1 mailcommon/filter/filteractioncopy.cpp
M +7 -0 mailcommon/filter/filteractioncopy.h
M +6 -0 mailcommon/filter/filteractiondelete.cpp
M +1 -0 mailcommon/filter/filteractiondelete.h
M +5 -0 mailcommon/filter/filteractionexec.cpp
M +1 -0 mailcommon/filter/filteractionexec.h
M +5 -0 mailcommon/filter/filteractionforward.cpp
M +1 -0 mailcommon/filter/filteractionforward.h
M +6 -0 mailcommon/filter/filteractionmove.cpp
M +2 -0 mailcommon/filter/filteractionmove.h
M +6 -0 mailcommon/filter/filteractionpipethrough.cpp
M +1 -0 mailcommon/filter/filteractionpipethrough.h
M +6 -0 mailcommon/filter/filteractionplaysound.cpp
M +2 -0 mailcommon/filter/filteractionplaysound.h
M +5 -0 mailcommon/filter/filteractionredirect.cpp
M +1 -0 mailcommon/filter/filteractionredirect.h
M +6 -1 mailcommon/filter/filteractionremoveheader.cpp
M +1 -0 mailcommon/filter/filteractionremoveheader.h
M +6 -0 mailcommon/filter/filteractionreplyto.cpp
M +1 -0 mailcommon/filter/filteractionreplyto.h
M +6 -0 mailcommon/filter/filteractionrewriteheader.cpp
M +1 ...

Read more...

Revision history for this message
In , Aaronw-x (aaronw-x) wrote :

I really hope this does get backported. The duplicate emails is a deal-killer for me and I'd LOVE to dump Thunderbird.

Revision history for this message
In , Andras Mantia (amantia) wrote :
Download full text (5.1 KiB)

Git commit e3a03f1d704838a9ced4ff41358785392774ae69 by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'KDE/4.9'.

(Backport filter fix from master)

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364
(cherry picked from commit d64d61b470c56b92294f6adee6d74305a217628d)

FIXED-IN: 4.9

M +7 -11 kmail/kmcommands.cpp
M +10 -7 kmail/kmcommands.h
M +1 -1 kmail/kmmainwidget.cpp
M +1 -5 mailcommon/filter/filteraction.cpp
M +6 -5 mailcommon/filter/filteraction.h
M +6 -0 mailcommon/filter/filteractionaddheader.cpp
M +2 -0 mailcommon/filter/filteractionaddheader.h
M +2 -2 mailcommon/filter/filteractionaddtag.cpp
M +1 -1 mailcommon/filter/filteractionaddtag.h
M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp
M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h
M +7 -0 mailcommon/filter/filteractionbeep.cpp
M +1 -0 mailcommon/filter/filteractionbeep.h
M +18 -1 mailcommon/filter/filteractioncopy.cpp
M +7 -0 mailcommon/filter/filteractioncopy.h
M +6 -0 mailcommon/filter/filteractiondelete.cpp
M +1 -0 mailcommon/filter/filteractiondelete.h
M +5 -0 mailcommon/filter/filteractionexec.cpp
M +1 -0 mailcommon/filter/filteractionexec.h
M +5 -0 mailcommon/filter/filteractionforward.cpp
M +1 -0 mailcommon/filter/filteractionforward.h
M +6 -0 mailcommon/filter/filteractionmove.cpp
M +2 -0 mailcommon/filter/filteractionmove.h
M +6 -0 mailcommon/filter/filteractionpipethrough.cpp
M +1 -0 mailcommon/filter/filteractionpipethrough.h
M +6 -0 mailcommon/filter/filteractionplaysound.cpp
M +2 -0 mailcommon/filter/filteractionplaysound.h
M +5 -0 mailcommon/filter/filteractionredirect.cpp
M +1 -0 mailcommon/filter/filteractionredirect.h
M +6 -1 mailcommon/filter/filteractionremoveheader.cpp
M +1 -0 mailcommon/filter/filteractionremoveheader.h
M +6 -0 mailcommon/filter/filteractionreplyto.cpp
M +1 ...

Read more...

Revision history for this message
In , Alexandre-bonneau (alexandre-bonneau) wrote :

Nice job !
/me is eager to try 4.9

Revision history for this message
In , Andras Mantia (amantia) wrote :

Git commit 8846d44b217108351c35f8afd883f72ee88388ec by Andras Mantia.
Committed on 06/07/2012 at 11:12.
Pushed by amantia into branch 'KDE/4.8'.

Backport part of the filter fix: this is the actual fix, the rest were optimizations and safety checks, but it is difficult to backport.
An unwanted side-effect might be that more data is downloaded than before,
based on the type of filters used.

FIXED-IN: 4.8.5
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364

M +0 -1 mailfilteragent/mailfilteragent.cpp

http://commits.kde.org/kdepim/8846d44b217108351c35f8afd883f72ee88388ec

Changed in kdepim:
status: Confirmed → Fix Released
Changed in kdepim:
status: Fix Released → Confirmed
172 comments hidden view all 252 comments
Revision history for this message
In , Alexandre-bonneau (alexandre-bonneau) wrote :

Kmail version 5.2.3 (in Debian Stable) is definitely showing this bug (see the attached screenshot).

Revision history for this message
In , Alexandre-bonneau (alexandre-bonneau) wrote :

Created attachment 107642
Duplicated mails in Kmail 5.2.3

Revision history for this message
In , Kde-v (kde-v) wrote :

Just hit the same. I noted that the duplicated messages, were messages never fully moved in the first place. They ghosted in the original folder, and the ghosts were copied.

The folders this happens to also does not have working folder properties. Nothing you change in folder properties have ANY effect and are immediately lost

1 comments hidden view all 252 comments
Revision history for this message
In , Árpád Lukács (lukacs-arpad) wrote :

With upgrading to version 5.5.3 on Kmail, the duplication problem got worse in my case.
I have a rather large (~ 10000) collection of messages. It happens after filtering, it can be "fixed" by clearing akonadi cache with akonadiconsole, but it appears again.
Adding a filter on top of the filter list, that executes "/usr/bin/sleep 15" did not help. I could not find out if it happens with the folders that contain many messages, or in all of them (i think the latter).
What did change with the new version (compared to the one there was in kubuntu 14.04) is that now I get an error message when trying to access the folder with the duplicate messages, that kmail cannot fetch the message because of multiple candidates. Previously, i could erase one.

Revision history for this message
In , DeMus (jan-mussche) wrote :

I can confirm this bug with both an IMAP and a POP account. Mails are sometimes (not always, filters don't always work well) transferred from the inbox to the correct folder and a little bit later the mails are also back in the inbox. This can go on forever.
When I delete them from the inbox it is over and they don't appear anymore.

Revision history for this message
In , RogerL (roger-larsson) wrote :

I have also seen this (and my wife on her setup - similar to mine)
and I wonder if some of this is due to the huge number of options and the more complex environment.

I want to be able to read mails both on my computer and on my mobile.
I want the computer to be the primary storage of my mails.
POP/IMAP account - using IMAP for both

Computer
- copied new messages locally using filters into correct folder
- filters that mark a message, resulted in two copies (one shadow - not readable) the updated state could be seen from mobile too...
- had "download for offline processing" enabled (trying with it off now)
   noticed that it
- push on
- recheck 5 minutes
...

I wonder if my settings caused numerous rereads, store on disk with different generations of each mail...

-rw-r--r-- 1 roger users 4589 11 mar 00.20 /home/roger/.local/share/akonadi/file_db_data/99/822899_r0
-rw-r--r-- 1 roger users 4100 6 apr 18.29 /home/roger/.local/share/akonadi/file_db_data/99/827899_r61
-rw-r--r-- 1 roger users 6372 6 apr 18.29 /home/roger/.local/share/akonadi/file_db_data/99/828199_r1360
-rw-r--r-- 1 roger users 4348 6 apr 18.30 /home/roger/.local/share/akonadi/file_db_data/99/831799_r1359
-rw-r--r-- 1 roger users 4126 6 apr 18.30 /home/roger/.local/share/akonadi/file_db_data/99/833599_r61

Revision history for this message
In , Psychonaut (psychonaut) wrote :

Is this problem still reproducible in the latest version of KMail? There haven't been any comments on this bug in over a year; I'm wondering if that's because the problem has been fixed or because it has driven most of the user base away. I'd love to return to using KMail but can't as long as this message duplication problem persists. (I was getting dozens of duplicated messages every day, which was too much to put up with.)

Revision history for this message
In , 3-eric-x (3-eric-x) wrote :

(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail? There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away. I'd love to return to using KMail but can't as long as
> this message duplication problem persists. (I was getting dozens of
> duplicated messages every day, which was too much to put up with.)

It has been fixed I believe, since it disappeared from my kmail in 2015.

Revision history for this message
In , Psychonaut (psychonaut) wrote :

(In reply to Eric Depagne from comment #203)

> It has been fixed I believe, since it disappeared from my kmail in 2015.

And yet there are plenty of comments here from 2015 to 2018 claiming the problem still exists. I would prefer to hear from someone who was experiencing the problem more recently (say, 2018) who can confirm that it is no longer reproducible.

Revision history for this message
In , Pe2er (pe2er) wrote :

(In reply to Eric Depagne from comment #203)
> (In reply to Tristan Miller from comment #202)
> > Is this problem still reproducible in the latest version of KMail? There
> > haven't been any comments on this bug in over a year; I'm wondering if
> > that's because the problem has been fixed or because it has driven most of
> > the user base away. I'd love to return to using KMail but can't as long as
> > this message duplication problem persists. (I was getting dozens of
> > duplicated messages every day, which was too much to put up with.)
>
> It has been fixed I believe, since it disappeared from my kmail in 2015.

It certainly hasn't. I still get numerous duplicates daily on this Gentoo box.

I think the comments dried up because (a) there was nothing new to add; (b) users grew tired of banging on to no avail.

Revision history for this message
In , Árpád Lukács (lukacs-arpad) wrote :

I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3), that came with Ubuntu 18.04. Then I started using the workaround from comment #118.

Revision history for this message
In , Psychonaut (psychonaut) wrote :

(In reply to Árpi from comment #206)
> I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3),
> that came with Ubuntu 18.04. Then I started using the workaround from
> comment #118.

Lucky you; that workaround did nothing for me.

Is there some sort of bug bounty program for KDE? If so I wouldn't mind chipping in €20 or so towards a fix for this issue.

Revision history for this message
In , Graeme Hewson (ghewson) wrote :

I haven't seen the problem for a while, maybe two years. I use POP3 only, not IMAP. I run Kubuntu; currently I'm on 19.04.

Sometimes kmail crashes, and I wonder if the way I clean up is related to my not seeing duplicate filtered messages. When kmail crashes, I stop akonadi ("akonadictl stop") and look in the maildir structure like this:

cd .local/share/.local-mail.directory
ls */new/*S

The S indicates messages which have been Seen, and according to the Maildir specification the files should not be in a"new" subdirectory; they should have been moved to a "cur" subdirectory. I therefore move them while akonadi is stopped. E.g.,

mv -i wastebin/new/*S wastebin/cur

I then run "akonadictl fsck" and restart akonadi ("akonadictl start").

As I say, it's pure speculation that this is why I don't see duplicate messages now.

Revision history for this message
In , Graeme Hewson (ghewson) wrote :

Apologies.

akonadictl start
akonadictl fsck

is the correct order.

Revision history for this message
In , Alexandre-bonneau (alexandre-bonneau) wrote :

On my end, I'm currently using Kmail 5.9.3 (Debian buster), and am seeing no duplicates using imap only.
Some very rare times I can see a few duplicates, but they disappear by themselves pretty quicly.

Revision history for this message
In , Jwagner-5 (jwagner-5) wrote :

(In reply to Tristan Miller from comment #202)
> [...] because the problem has been fixed or because it has driven most of
> the user base away. I'd love to return to using KMail [...]

I didn't stop using KMail. I stopped using filters.

If I find time to explore alternatives and find something suitable I may switch. Also possible I will use two mail clients in parallel.

Coincidentally, I am well positioned to test alternatives as I moved away from using folders under "Local Folders" to using a local imap server a while ago (because, at that time, KMail stored messages in a database instead of a maildir). This should make it easy to use multiple clients in parallel.

Revision history for this message
In , S-pamme-f (s-pamme-f) wrote :

(In reply to Eric Depagne from comment #203)
> (In reply to Tristan Miller from comment #202)
> > Is this problem still reproducible in the latest version of KMail? There
> > haven't been any comments on this bug in over a year; I'm wondering if
> > that's because the problem has been fixed or because it has driven most of
> > the user base away. I'd love to return to using KMail but can't as long as
> > this message duplication problem persists. (I was getting dozens of
> > duplicated messages every day, which was too much to put up with.)
>
> It has been fixed I believe, since it disappeared from my kmail in 2015.

Most definitely not fixed. Still run into it multiple times per day with some messages showing up 5 or more times. Sometimes I just delete the entire folder and filter and just set everything up again. Sometimes seems to do the trick. I have been unable to find any logic behind why some messages get duplicated where others don't (in the same folder).

Revision history for this message
In , Árpád Lukács (lukacs-arpad) wrote :

(In reply to Tristan Miller from comment #207)
> (In reply to Árpi from comment #206)
> > I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3),
> > that came with Ubuntu 18.04. Then I started using the workaround from
> > comment #118.
>
> Lucky you; that workaround did nothing for me.
>
> Is there some sort of bug bounty program for KDE? If so I wouldn't mind
> chipping in €20 or so towards a fix for this issue.

I checked my filters; what I am doing is not exactly as described in comment #118, but instead, I added an extra action, to execute
/bin/sleep 1
to each of my filters. Since then, no duplicates.

Revision history for this message
In , Tobias-leupold-u (tobias-leupold-u) wrote :

I've been using IMAP for some years now, and I don't get duplicates anymore when filtering. In the case my filters ever do filter, which is roughly the case in 80 % of the mails they should work on (otherwise, nothing happens).

After almost eight years and more than 200 comments, I think one should rather wonder if this even can be fixed inside this Akonadi monster I personally wouldn't even dare to look at the code, or if one should reconsider the whole thing.

Only thing I can't understand is that the KMail guys are not annoyed themselves by this ... I mean, I don't think they use Trojitá, do they?! But no harm meant,
remarkably enough I still stick to KMail ;-)

Revision history for this message
In , Bingmybong (bingmybong) wrote :

(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail? There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away. I'd love to return to using KMail but can't as long as
> this message duplication problem persists. (I was getting dozens of
> duplicated messages every day, which was too much to put up with.)

The way i reduced the duplicates to a point where it hardly happens (maybe once a week) is to stop filtering on incoming messages and only apply the filters manually once all emails are in. I'm using IMAP but I think the POP3 still has a lot of issues.

Revision history for this message
In , Bll-guayam-gj5 (bll-guayam-gj5) wrote :

(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail? There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away.

Driven away.

Was a KMail for what seems like forever (~2000?) (I've been a KDE user since it first came out in '96), but after dealing with workarounds and hacks for two+ years I eventually gave in and abandoned KMail back in mid-2017.

Took me multiple weeks to get all of my mail (~12GB) imported into Thunderbird, get my 200+ mail filters set up again, and to remove all of the duplicates and junk that various KMail bugs had done to my mail. Was the most painful thing I'd done in years, and I knew the switch was going to suck bad so I held out as long as I could.

KMail eventually just got unusable, period. I'd have to do workarounds and restart akonadi every single time I picked up my POP mail, which just made KMail effectively useless. After realizing I was using Alpine to read all of my new mail and delete junk _before_ fetching it into KMail, I finally gave in and switched.

LOVED KMail, but THIS bug led me to finally throw in the towel and give it up.

Revision history for this message
In , Ederag (ederag) wrote :

(In reply to graham from comment #216)
> Was a KMail for what seems like forever (~2000?) (I've been a KDE user since
> it first came out in '96), but after dealing with workarounds and hacks for
> two+ years I eventually gave in and abandoned KMail back in mid-2017.

There was a commit in 2017-05-04
https://phabricator.kde.org/R206:43f2cde61f98317eb13d98222a57bc6df323a308
that was about this bug.
And indeed in openSUSE Leap-15 (kdepim-runtime 17.12.3),
there are much fewer duplicates than before.

There is now a related bug about unreachable duplicates
that has already been reported as #388869.

(In reply to Tobias Leupold from comment #214)
> I think one should rather wonder if this even can be fixed
> inside this Akonadi monster I personally wouldn't even dare to look at
> the code, or if one should reconsider the whole thing.

kmail/akonadi handle my > 15GB maildir gracefully,
which is quite a performance. Congratulations !

So it is not perfect, but to me it is an amazing piece of software.
Please consider that it is very hard for maintainers
to fix a bug that is not reproducible on their setup.

Revision history for this message
In , Ederag (ederag) wrote :

With a working link: bug #388869

Revision history for this message
In , Pe2er (pe2er) wrote :

> Please consider that it is very hard for maintainers
> to fix a bug that is not reproducible on their setup.

True, but there's nothing to stop them from enlisting help from knowledgeable users (at least a few have offered, including me). They won't do it though; I can only speculate as to why not.

Revision history for this message
Tom (thomasmca) wrote :

This bug is happening to me using KMail 5.10.3 on Lubuntu 19.04. Four emails that get moved into a folder via a filter get downloaded into that folder over and over. I deleted 20+ copies of each message recently, and now there are 8 more of each one.

I am downloading via POP3 from GMail.

Revision history for this message
In , Linuxfan-2 (linuxfan-2) wrote :

Version KMail 5.13.3 (19.12.3)
OS Gentoo Linux

I have been plagued by this bug for a very long time. Today I found out, that for ages there has been an error message in my mysql/mariadb log-file:

[ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set ...

This is the first time I saw this error, as there is no other application than akonadi using mariadb on my pc and akonadi was never complaining. This mariadb-error could be fixed by running mysql_upgrade (google it to find an explanation, this is due to an incomplete upgrade of the mariadb-package)

After fixing the database, kmail has not been showing duplicated mails any longer. Also filters have been working flawlessly.

Revision history for this message
In , Ederag (ederag) wrote :

(In reply to linuxfan from comment #220)
> for ages there has been an error message in my mysql/mariadb log-file:
>
> [ERROR] Incorrect definition of table mysql.event: expected column
> 'sql_mode' at position 14 to have type set ...

Thanks for sharing.
In which log file exactly did you find this error ?

/var/log/mysql/ is empty here, so

pgrep -af mysqld
/usr/sbin/mysqld --defaults-file=/home/ederag/.local/share/akonadi/mysql.conf --datadir=/home/ederag/.local/share/akonadi/db_data/ --socket=/tmp/akonadi-ederag.Yqlbg1/mysql.socket --pid-file=/tmp/akonadi-ederag.Yqlbg1/mysql.pid

grep -1 "error log" /home/ederag/.local/share/akonadi/mysql.conf

# # error log file name, relative to datadir (default:hostname.err)
log_error=mysql.err

grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
<nothing>

That was with the "1 s" delay filter active,
just deactivated; we'll see.

Revision history for this message
In , Pe2er (pe2er) wrote :

(In reply to ederag from comment #221)

> grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
> <nothing>

Here, I see:

$ grep -i error .local/share/akonadi/db_data/mysql.err
2020-05-08 13:12:10 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2020-05-08 13:12:10 9 [ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not found.

I don't know how long that's been happening, because I haven't had any duplicates for quite a long time. I remember, though, that when I was getting them, that error log had numerous errors of various types.

Revision history for this message
In , Bingmybong (bingmybong) wrote :

(In reply to Peter Humphrey from comment #222)
> (In reply to ederag from comment #221)
>
> > grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
> > <nothing>
>
> Here, I see:
>
> $ grep -i error .local/share/akonadi/db_data/mysql.err
> 2020-05-08 13:12:10 0 [ERROR] Can't open and lock privilege tables: Table
> 'mysql.servers' doesn't exist
> 2020-05-08 13:12:10 9 [ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not
> found.
>
> I don't know how long that's been happening, because I haven't had any
> duplicates for quite a long time. I remember, though, that when I was
> getting them, that error log had numerous errors of various types.

I've been using postgresql and i've hardly seen any since but as of today i got duplicates by the bucket load.

opensuse:tumbleweed:20200506
Qt: 5.14.1 KDE Frameworks: 5.69.0 - KDE Plasma: 5.18.4 - kwin 5.18.4
kmail2 5.14.0 (20.04.0) - akonadiserver 5.14.0 (20.04.0) -

Revision history for this message
In , Linuxfan-2 (linuxfan-2) wrote :

the log file is /var/log/mysql/mysqld.err

However, digging deeper, the problem might be related to two instances of mariadb being present.

pgrep -af mysqld yields

2402 /usr/sbin/mysqld --defaults-file=/etc/mysql/my.cnf
2881 /usr/sbin/mysqld --defaults-file=/home/linuxfan/.local/share/akonadi/mysql.conf --datadir=/home/linuxfan/.local/share/akonadi/db_data/ --socket=/run/user/10001/akonadi/mysql.socket --pid-file=/run/user/1000/akonadi/mysql.pid

While /var/log/mysql/mysqld.err is now clear, in

/home/linuxfan/.local/share/akonadi/db_data/mysql.err

I can see two further errors and some warnings:

[ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
 [Warning] Can't open and lock time zone table: Table 'mysql.time_zone_leap_second' doesn't exist trying to live without them
 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
 [Note] Reading of all Master_info entries succeeded
 [Note] Added new Master_info '' to hash table
 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.4.12-MariaDB' socket: '/run/user/1000/akonadi/mysql.socket' port: 0 Source distribution
[ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not found.

what is more important, I believe, is that akonadi seems to start the second mysql instance but some of the data is actually going to my main mysql server instead and its data directory /home/mysql_data/ while other parts are ending up in the akonadi-instance of mysql.

After disabling the stand-alone server, I found that ~/.config/akonadi/akonadiserverrc

still contained the line

Options="UNIX_SOCKET=/var/run/mysql/mysqld.socket"

pointing to the main server socket. Clearing this up, everything seems to end up in the akonadi-version of mysql (with the above warnings/errors, not sure what that means). I am unsure, why some of the data still went to the akonadi-version of mysql, though.

From my point of view, I seem to have had some mess in the config files as I have been dragging/upgrading my /home/ directory with me since 1997 .... (and yes, I have been using KDE ever since the original beta1)

Revision history for this message
In , Ederag (ederag) wrote :

Well, only half a day before a duplicate is back.
(again, from lists.gnu.org)
Same status as in comment #221.
And nothing suspicious in mysql.err (no ERROR).

Revision history for this message
In , Alexandre-bonneau (alexandre-bonneau) wrote :

(In reply to Alexandre Bonneau from comment #210)
> On my end, I'm currently using Kmail 5.9.3 (Debian buster), and am seeing no
> duplicates using imap only.
> Some very rare times I can see a few duplicates, but they disappear by
> themselves pretty quicly.

For info (I posted that info in another related issue already), I found out that I only get duplicates when I have 2 computers running akonadi-server.

For instance it happens when I'm on my main rig, then casually open my laptop. Then mails are duplicated a lot...until I close one of the akonadi server (usually the one from the laptop).

Revision history for this message
In , Ederag (ederag) wrote :

(In reply to Alexandre Bonneau from comment #226)
> For info (I posted that info in another related issue already), I found out
> that I only get duplicates when I have 2 computers running akonadi-server.

Here also, two computers once concurrently ran akonadi
with the same configuration (synchronized with unison).
But duplicates do appear, even when the other akonadi is stopped.
(pgrep akonadi yields nothing on the other computer)
[akonadictl 5.16.1 (20.12.0)]

Revision history for this message
In , R-kunschke (r-kunschke) wrote :

The problem is still present in kmail for a long time. I have filters for using bogofilter in kmail.
Is there any workaround or something? Or a filter for kmail which removes all duplicates?
And is there any explanation why this is happenning?

Revision history for this message
In , Ederag (ederag) wrote :

(In reply to r.kunschke from comment #228)
> Is there any workaround or something?

Adding this filter helped a lot here:
- match all messages
- Executed Command: /usr/bin/sleep 1,

Revision history for this message
In , Christian Hilberg (hilberg) wrote :

Created attachment 148406
signature.asc

Am Mittwoch, 27. April 2022, 10:57:26 CEST schrieb ederag:
> https://bugs.kde.org/show_bug.cgi?id=283682
>
> --- Comment #229 from ederag <email address hidden> ---
> (In reply to r.kunschke from comment #228)
> > Is there any workaround or something?
>
> Adding this filter helped a lot here:
> - match all messages
> - Executed Command: /usr/bin/sleep 1,

Depending on how quick the IMAP server is, this had
little to no effect for me when I tried, so YMMV.

(bye)^2

Revision history for this message
In , Bingmybong (bingmybong) wrote :

(In reply to r.kunschke from comment #228)
> The problem is still present in kmail for a long time. I have filters for
> using bogofilter in kmail.
> Is there any workaround or something? Or a filter for kmail which removes
> all duplicates?
> And is there any explanation why this is happenning?

There is an option on the "Folder" menu to remove duplicates that works reasonably well.
I used to have this issue but its now pretty much gone since i changed from 1. POP3 to Imap and 2. Mysql to Postgresql for the backend, its very very rare for me to get a duplicate now

Revision history for this message
In , Christoph Thielecke (crissi99) wrote :

Still valid in akonadi 21.12.3 on Kubuntu 22.04.

I think akonadi should automatically handle this problem.

Revision history for this message
In , Pe2er (pe2er) wrote :

New symptom.

I have a local server that fetches POP3 mail from my ISP and serves it as IMAP4. I've had only rare duplicates since setting this arrangement up a few years ago.

I discovered yesterday that creating a KMail POP3 account led immediately to numerous duplicates, as half-expected, but the key point here is that duplicates kept occurring, even after I'd deleted the POP3 account and rebooted.

This is KMail version 5.20.3 (22.04.3) on Gentoo stable, Plasma version 5.24.6-r1. I tried postgres several years ago but it didn't improve anything, so I'm sticking with mariadb.

Displaying first 40 and last 40 comments. View all 252 comments or add a comment.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.