KMail duplicates emails

[Volodymyr Kolesnykov]

KMail duplicates emails from time to time.

Please see the screen shot attached.

This happens every day, several times a day and is really annoying.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: kmail 4:4.7.2+git111007-0ubuntu1
ProcVersionSignature: Ubuntu 3.0.0-12.20-server 3.0.4
Uname: Linux 3.0.0-12-server x86_64
NonfreeKernelModules: fglrx
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
Date: Sat Oct 29 15:23:06 2011
ExecutablePath: /usr/bin/kmail
ProcEnviron:
 LANGUAGE=ru_RU:ru
 LANG=ru_RU.UTF-8
 SHELL=/bin/bash
SourcePackage: kdepim
UpgradeStatus: Upgraded to oneiric on 2011-08-21 (68 days ago)

[Maednoldor (maednoldor)]

Version: 4.7 (using Devel)
OS: Linux

I have a Jira account. I have set up 2 filters: one is used to move messages for JIRA folder and mark it as read. Filtered by sender (my own activieties).
The second one is used to move all other JIRA maessages to the same folder, but do not mark them as read.

As a result, I have a lot of duplicated messages in my JIRA folder. Some messages (during 2-3 weeks) became duplicated to 10 times.

Reproducible: Sometimes

Steps to Reproduce:
1. create some filters, as I mentioned.
2. try to get the mail.

Actual Results:
New messages arrive in the INBOX folder (that are already placed to the JIRA folder). After this, KMail starts to filter them and send the duplicates to JIRA folder again.

Expected Results:
Messages should be placed to the folder. No duplicating.

As I remember, one of e-mail clients, used by mine, has a function 'remove duplicates'. This should be very useful in this case.

[Alessandro Menti (elgaton)]

Thanks for reporting the bug.

Could you please specify in more detail when the duplicates start appearing? For example, are they already there when you start the program, after you move/copy some messages... or does this bug appear randomly? If possible, could you provide a test case (a sequence of actions that can be performed to reproduce the bug)?

[Volodymyr Kolesnykov (sjinks)]

From what I can tell, this has to do something with filters - as it happens only on accounts I have set up filters for (e.g., if "Anywhere in Header" contains "X-Launchpad-Hash", move email into "KMail Folders/Launchpad").

Emails get duplicated usually when new mail arrives.

Sometimes emails resurrect after I delete them.

This makes me think that when email is moved by the filter rule, KMail does not actually delete it - just hides it. Then when new mail arrives, it finds that "semi-deleted" email, applies a filter to it and now I get a duplicate of the existing emails.

I often see a warning "KMail Folders: Cannot modify emails in folder" (the warning is in Russian so my translation is approximate).

[Volodymyr Kolesnykov (sjinks)]

Another example: I received these emails: https://url.odesk.com/u2fi4 (screenshot)

I read them and deleted them.

Then on the next mail check I see this: https://url.odesk.com/7t69v (screenshot)

That new email is the one I have deleted earlier.

The problem is that it does not happen everytime with every email - but often enough to be very annoying.

[Alessandro Menti (elgaton)]

(Reassigned the bug to the right package.)

Are you perhaps using an IMAP server? I have run a search on the KDE bugtracker and found the following bug:
<https://bugs.kde.org/show_bug.cgi?id=95064> - Duplicates messages in imap folder after filtering
It is related to an old version of KMail, but it seems (at least from the last comments) that it occurs on some newer versions as well.
If so, could you please tell me so I can link this bug report to the one on the KDE bugtracker?

[Volodymyr Kolesnykov (sjinks)]

The problem occurs only on GMail's POP3 account (actually that's the only account that uses filters).

Two more observations:
* only filtered emails get duplicated/resurrected (e.g., only when emails were moved from one folder to another);
* this only happens when the filter has to process several emails at once (I do not remember seeing this behavior when only one email comes - only when there are multiple emails).

[Volodymyr Kolesnykov (sjinks)]

One more example:

1. 10 emails have come (screenshot: https://url.odesk.com/kze6s ). i have read and deleted them.
2. 30 minutes later (or so) when new emails came: https://url.odesk.com/njyqc

monit alerts (both), lfd on snowboarding-mail-0 and abuse report (basically 4 of 5 emails KMail notified me about) are "resurrected" emails - they appeared after I deleted them.

[Alessandro Menti (elgaton)]

OK - I think that this other bug report (<https://bugs.kde.org/show_bug.cgi?id=283682> - KMail duplicates filtered messages) matches both the problem and the version of KMail you are using. (It may be a bit difficult to reproduce the bug since it partially happens at random - I'm referring to the deletion part).

If you find that the bug report matches your problem too (I'm not 100% sure), could you please leave a comment in the KDE bugtracker (providing a link to this bug report); otherwise, could you please open a new bug report at the KDE bugtracker (<https://bugs.kde.org/>), filing it against the product "kmail2" and the version "4.7"? This way, the KMail developers will be notified of the bug and will fix it.

In any case, after you have completed the process, please click on the link "Also affects project" at the top of this bug report and write the URL of the bug in the KDE bugtracker there. This way, we can notice when the bug has been fixed by the KMail developers and release a new version of KMail in Ubuntu accordingly.

Thanks again for all the information!

[Volodymyr Kolesnykov (sjinks)]

Could be the same bug: https://bugs.launchpad.net/ubuntu/+source/kdepim/+bug/883459

[Volodymyr Kolesnykov (sjinks)]

Done.

[Alessandro Menti (elgaton)]

Thanks again!

[Montel-3 (montel-3)]

*** Bug 286053 has been marked as a duplicate of this bug. ***

[Geoff Clements (geoff-v8x)]

I have the same problem, nearly all messages in my inbox are duplicated. I use only IMAP so don't know if this affects pop3 accounts. I do have filtering enabled.

[Rick Vernam (rickv-hobi)]

Running Gentoo stable, just pulled in kmail2. my pop account is showing multiple copies of messages that have been moved to another folder by a filter rule. Also, when I click on the Inbox messages that had been moved by filter suddenly appear in the inbox, then are moved back into the the folder specified by the filter rule (and thus the duplicate).
Also checking email will cause the filtered messages to duplicate.
This is not happening with my IMAP accounts (though none of my filters affect messages that typically come in through the accounts I access via IMAP).

Happy hunting.

[Rick Vernam (rickv-hobi)]

I exported all my filters, and deleted them from KMail2. KMail2 is now running without filters. If I move an email from the inbox to another folder (ie, the "destination" folder), the email will appear again in the inbox (and remain in the destination folder as well). This occurs both when the destination folder is a sub-folder of the inbox, and when the destination folder is directly under the Local Folders.

[Pragma-n (pragma-n)]

I experience this problem in 4.7.90 (4.8 beta2).

If a mail just downloaded is moved by a filter to a different folder then there is a good chance that it will re-appear again in the original folder at the next update

It does not happen with all the accounts or all the filtered messages. However if a message is affected then it's 100% reproduceable: it will be duplicated at every folder update (forever?).

[Ggrau (ggrau)]

Same here... with 4.7.4 on Fedora16
Most mails I try to move from my Inbox will appear again as duplicate.
This happens also, when moved by filter rule.
Pressing <Del> to move them into trash surprisingly seems to work.

[Matthias-nagel-3 (matthias-nagel-3)]

Same problem for me. I am using kmail 4.7.4-r1 from the Gentoo Stable amd64 repository. I have to POP3 accounts and some filters that examine the mail header (mostly the "from" attribute) and move the mails into folders and marks them as read. Sometimes new messages are duplicated. This also affects mails that are passed through the filter but not modified, because the filter condition does not apply.

The mails are not duplicated immediately but reemerge after a folder update, especially after starting kmail. Suddenly mails that are already some days old and have already been processed by the filter show up again.

I can observe the same behaviour as in comment #6. Not every mail that is processed by some filter is affected, but if some mail is affected once, the mail reappears over and over again. I have the feeling that a mail is more likely to be affected if the filter has to process a lot of new mails at once.

[Tokoe (tokoe)]

Hej all,

can you please check whether akonadi reports some errors regarding the target folders you try to move the mails to, about a missing remote id?
You can also use akonadiconsole and check the folder properties on the 'internal' tab to find out the remote id.

[Matthias Mailänder (mailaender)]

Found this while trying to reproduce and understand akonadiconsole logs:

AgentBase(akonadi_imap_resource_0): Select failed, server replied: A000068 NO Mailbox doesn't exist: &APw-lleimer

The local Trash folder is called "Lokale Ordner/Mülleimer". This is where the spam check wants to move spam. Putting it on the IMAP Junk folder as Thunderbird does is not possible/allowed.

[Geoff Clements (geoff-v8x)]

(In reply to comment #9)
> Hej all,
>
> can you please check whether akonadi reports some errors regarding the target
> folders you try to move the mails to, about a missing remote id?
> You can also use akonadiconsole and check the folder properties on the
> 'internal' tab to find out the remote id.

Tobias,

Sorry this will sound lame but I'm not sure where to look for the akonadi error log. I did try this:

$ grep -R remote ~/.local/share/akonadi/
grep: /home/geoff/.local/share/akonadi/socket-dhcppc2: No such file or directory
grep: /home/geoff/.local/share/akonadi/socket-dhcppc1: No such file or directory
Binary file /home/geoff/.local/share/akonadi/db_data/ibdata1 matches
Binary file /home/geoff/.local/share/akonadi/db_data/akonadi/pimitemtable.frm matches
Binary file /home/geoff/.local/share/akonadi/db_data/akonadi/collectiontable.frm matches
Binary file /home/geoff/.local/share/akonadi/db_data/ib_logfile0 matches
grep: /home/geoff/.local/share/akonadi/akonadiserver.socket: No such device or address
grep: /home/geoff/.local/share/akonadi/socket-scotty/akonadiserver.socket: No such device or address
grep: /home/geoff/.local/share/akonadi/socket-scotty/mysql.socket: No such device or address

As the only matches are in binary files I didn't take this any further.

Using akonadiconsole there is nothing at all displayed in the browser->internals tab. I do notice that non of the remote IMAP folders are displayed in the panel on the left hand side. I see only:

Personal Contacts
Local Folders
    |
     - Outbox
Search

[Ggrau (ggrau)]

All I can see when moving a Mail is:

==================================
NotificationManager::notify ( Item (129274, 1329071216.R951.neptun:2,S) from collection 467 movedto collection 535 )
==================================

When rescanning the derectory, I just moved it out with F5:
========================================
NotificationManager::notify ( Item (129275, 1329071216.R951.neptun:2,S) in collection 467 added )
NotificationManager::notify ( Item (129275, 1329071216.R951.neptun:2,S) in collection 467 modified parts (PLD:RFC822) )
=======================================

There is a lot of debug traffic in the upper window, but (as I can see) nothin g suspecious.

[Milian Wolff (milianw)]

Tobias, please give us more information on what exactly we should look at and post here.

[Milian Wolff (milianw)]

rightclick on (some) of my target folders in akonadiconsole shows that they all have a "remote identifier" set.

I note though, that the situation has improved, I don't get as many duplicated emails anymore as I used to before. I'll see whehter I can reproduce it at all, maybe this is actually fixed in recent git?

[Christian Trippe (christiandehne)]

*** Bug 282296 has been marked as a duplicate of this bug. ***

[Geoff Clements (geoff-v8x)]

(In reply to comment #9)
> Hej all,
>
> can you please check whether akonadi reports some errors regarding the target
> folders you try to move the mails to, about a missing remote id?
> You can also use akonadiconsole and check the folder properties on the
> 'internal' tab to find out the remote id.

Tobias,

I have checked a number of times now in akonadiconsole, the debugger does not show any error messages when the messages are duplicated. Please note however that the duplicated messages are the ones that are /not/ moved because they do not meet any filter criteria. The messages that are moved by the filters are not duplicated.

[Matthias-nagel-3 (matthias-nagel-3)]

Opposed to comment #16 the mails that are moved are duplicated with me.

Even mails that are not moved by a filter but that are move manually (drag and drop) are duplicated. Mails that are "moved" to the bin folder by hitting "DEL" are duplicated, too. In all three cases the mail re-appears in the original folder after some period of time. The problem can be avoid by copying the mail to the destination folder and then purge the original mail from the source folder by hitting "Shift+DEL". This work-around also applies to the filter. If I create two filter actions - copy mail first, then purge the original - everything is fine.

And in all three cases there is no error report in the akonadiconsole.

[Milian Wolff (milianw)]

reopening, as I also just noticed this again. The target folder has a proper remote identifier set, yet I saw that some mails where duplicated by the filter manager...

Tobias, any other ideas what I could look into?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alexandre (ab-linuxfr)]

I have the same problem, but with 4.8.1.

[Alexandre-bonneau (alexandre-bonneau)]

This nasty bugs affect 4.8.1 too.

[Dr-bugzilla (dr-bugzilla)]

I believe the problem I have in 4.8.4 is the same problem: emails passing through filters are sometimes being duplicated.

I have a cron job on my servers which send me an "hourly test email" message which I receive on my desktop PC; these have been running for about two years without trouble. I have upgraded my desktop from Ubuntu 10.04 and kmail to Xubuntu 64-bit 11.10 and kmail v4.7.3 on my desktop and I am now seeing some of these hourly emails as identical duplicates (same ids in the headers, etc.) in their folders in kmail2.

For example, my PC was unattended for 11 hours last night and during that time 3 of the hourly emails appear as duplicates in their folder.

The emails are fetched by kmail2 using POP3 and filed into mbox folders by filters. When using kmail I had it set up to remove emails from servers; I set kmail2 to leave it on the servers (just in case I reverted back). On the server I can see there are no duplicates so they were not sent as duplicates.

The rule checks the subject of the email contains "Hourly Test Email" and the "Envelope-to" equals the email address I specified for this sending server. it then moves the email to the correct folder and processing rules is then stopped.

These email checking filters are applied before "Bogofilter check", "Spam Handling" "Classify as Spam" and "Classify as NOT spam" filters, so they are all probably innocent.

The email was downloaded and processed by "interval email checking".

To reproduce it:
1. Set up a filter to receiving incoming email and automatically file it in a folder.
2. Send lots of emails to it.
3. Check the folder.

I just did this by manually sending 18 emails; one of them in the middle was duplicated.

I set up a script to send an email every 5 seconds and sent 77 emails. 79 arrived; two had been duplicated.

The package I have installed is: kmail 4:4.7.4+git111222-0ubuntu0.1
"About KMail" says "KMail Version 4.7.3, Using KDE Development Platform 4.7.4 (4.7.4)"

[Alexandre-bonneau (alexandre-bonneau)]

Just for information, the bug is still seen in 4.8.2.

[PhobosK (phobosk)]

Affects heavily KDE/KMail 4.8.4 too...
The problem is definitely in moving messages by an applied filter. Manual filtering doesn't have it.
The affected messages are marked for deletion on the remote server (IMAP in my case), deleted from the KMail folder and then reappearing back in the same Kmail folder and in the remote folder too. So at some point in time, the remote folder has the marked for deletion messages + one fresh new and duplicate of all marked for deletion. The strange thing is that some of the duplicates have only headers and not the full messages.
So the problem seems to be in the Move action function of the filtering rules.

[Andras Mantia (amantia)]

Git commit d64d61b470c56b92294f6adee6d74305a217628d by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'master'.

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364

M +7 -11 kmail/kmcommands.cpp
M +10 -7 kmail/kmcommands.h
M +1 -1 kmail/kmmainwidget.cpp
M +1 -5 mailcommon/filter/filteraction.cpp
M +6 -5 mailcommon/filter/filteraction.h
M +6 -0 mailcommon/filter/filteractionaddheader.cpp
M +2 -0 mailcommon/filter/filteractionaddheader.h
M +2 -2 mailcommon/filter/filteractionaddtag.cpp
M +1 -1 mailcommon/filter/filteractionaddtag.h
M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp
M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h
M +7 -0 mailcommon/filter/filteractionbeep.cpp
M +1 -0 mailcommon/filter/filteractionbeep.h
M +18 -1 mailcommon/filter/filteractioncopy.cpp
M +7 -0 mailcommon/filter/filteractioncopy.h
M +6 -0 mailcommon/filter/filteractiondelete.cpp
M +1 -0 mailcommon/filter/filteractiondelete.h
M +5 -0 mailcommon/filter/filteractionexec.cpp
M +1 -0 mailcommon/filter/filteractionexec.h
M +5 -0 mailcommon/filter/filteractionforward.cpp
M +1 -0 mailcommon/filter/filteractionforward.h
M +6 -0 mailcommon/filter/filteractionmove.cpp
M +2 -0 mailcommon/filter/filteractionmove.h
M +6 -0 mailcommon/filter/filteractionpipethrough.cpp
M +1 -0 mailcommon/filter/filteractionpipethrough.h
M +6 -0 mailcommon/filter/filteractionplaysound.cpp
M +2 -0 mailcommon/filter/filteractionplaysound.h
M +5 -0 mailcommon/filter/filteractionredirect.cpp
M +1 -0 mailcommon/filter/filteractionredirect.h
M +6 -1 mailcommon/filter/filteractionremoveheader.cpp
M +1 -0 mailcommon/filter/filteractionremoveheader.h
M +6 -0 mailcommon/filter/filteractionreplyto.cpp
M +1 -0 mailcommon/filter/filteractionreplyto.h
M +6 -0 mailcommon/filter/filteractionrewriteheader.cpp
M +1 ...

[Aaronw-x (aaronw-x)]

I really hope this does get backported. The duplicate emails is a deal-killer for me and I'd LOVE to dump Thunderbird.

[Andras Mantia (amantia)]

Git commit e3a03f1d704838a9ced4ff41358785392774ae69 by Andras Mantia.
Committed on 04/07/2012 at 17:35.
Pushed by amantia into branch 'KDE/4.9'.

(Backport filter fix from master)

A biggish refactoring of the filter functionality:

1) Do not take the items from the cache only, as they might be incomplete (especially in online imap case, but not only).
This should fix the body loss bug on spam filtering.(Bug 287752 and similar ones + countless of reports on user list)
Might fix 293768.

2) Download only what is really needed for the filter and add safety checks that
the mail is not modified if the requested part is missing. So far the distinction
was only betwee the body and rest. This should speed up filtering.
Also related to 287752, but should fix also 292283, 288109.

3) Rename qDebug->kDebug + fix some includes

4) The code should also fix the duplicate mails appearing after filtering (some of the duplicates
with empty content), eg. 275233

All bug reporters: in case you still see the bug, reopen the corresponding one.
For the reports that were CC'd, but not closed, please try to reproduce and tell
the result.

So far the bug is fixed on master, but might be backported later to 4.9.
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364
(cherry picked from commit d64d61b470c56b92294f6adee6d74305a217628d)

FIXED-IN: 4.9

M +7 -11 kmail/kmcommands.cpp
M +10 -7 kmail/kmcommands.h
M +1 -1 kmail/kmmainwidget.cpp
M +1 -5 mailcommon/filter/filteraction.cpp
M +6 -5 mailcommon/filter/filteraction.h
M +6 -0 mailcommon/filter/filteractionaddheader.cpp
M +2 -0 mailcommon/filter/filteractionaddheader.h
M +2 -2 mailcommon/filter/filteractionaddtag.cpp
M +1 -1 mailcommon/filter/filteractionaddtag.h
M +9 -3 mailcommon/filter/filteractionaddtoaddressbook.cpp
M +2 -0 mailcommon/filter/filteractionaddtoaddressbook.h
M +7 -0 mailcommon/filter/filteractionbeep.cpp
M +1 -0 mailcommon/filter/filteractionbeep.h
M +18 -1 mailcommon/filter/filteractioncopy.cpp
M +7 -0 mailcommon/filter/filteractioncopy.h
M +6 -0 mailcommon/filter/filteractiondelete.cpp
M +1 -0 mailcommon/filter/filteractiondelete.h
M +5 -0 mailcommon/filter/filteractionexec.cpp
M +1 -0 mailcommon/filter/filteractionexec.h
M +5 -0 mailcommon/filter/filteractionforward.cpp
M +1 -0 mailcommon/filter/filteractionforward.h
M +6 -0 mailcommon/filter/filteractionmove.cpp
M +2 -0 mailcommon/filter/filteractionmove.h
M +6 -0 mailcommon/filter/filteractionpipethrough.cpp
M +1 -0 mailcommon/filter/filteractionpipethrough.h
M +6 -0 mailcommon/filter/filteractionplaysound.cpp
M +2 -0 mailcommon/filter/filteractionplaysound.h
M +5 -0 mailcommon/filter/filteractionredirect.cpp
M +1 -0 mailcommon/filter/filteractionredirect.h
M +6 -1 mailcommon/filter/filteractionremoveheader.cpp
M +1 -0 mailcommon/filter/filteractionremoveheader.h
M +6 -0 mailcommon/filter/filteractionreplyto.cpp
M +1 ...

[Alexandre-bonneau (alexandre-bonneau)]

Nice job !
/me is eager to try 4.9

[Andras Mantia (amantia)]

Git commit 8846d44b217108351c35f8afd883f72ee88388ec by Andras Mantia.
Committed on 06/07/2012 at 11:12.
Pushed by amantia into branch 'KDE/4.8'.

Backport part of the filter fix: this is the actual fix, the rest were optimizations and safety checks, but it is difficult to backport.
An unwanted side-effect might be that more data is downloaded than before,
based on the type of filters used.

FIXED-IN: 4.8.5
Related: bug 287752, bug 292283, bug 288109, bug 286043, bug 295484, bug 302337, bug 295000, bug 295684, bug 293768, bug 275233, bug 293918, bug 284310, bug 286364

M +0 -1 mailfilteragent/mailfilteragent.cpp

http://commits.kde.org/kdepim/8846d44b217108351c35f8afd883f72ee88388ec

[Christian Trippe (christiandehne)]

First, thanks for working on the filter problems!

Using KDE 4.8 with the minimal 4.8 patch I sometimes still see duplicated messages, but much less then before. And it looks like the duplicates now always have content.

I'll report back how 4.9 works for me.

[Marcus Gama (marcus-gama-gmail)]

I'm using KDE 4.9 and, at least for me, the issue was not fixed. I have duplicated messages in my Inbox.

[Aaronw-x (aaronw-x)]

It is even worse for me. Somehow it's mangling my email and nuking the body, leaving only the header in place. I had to nuke Akonadi.

[Alexandre-bonneau (alexandre-bonneau)]

This issue is indeed still affecting KDE 4.9.
It's worse as it happens much more than before, and instead of duplicating approximatively 12 times each mail, it seems it's more around 16 times now.

This is a killer for Kmail..

[rabauke (sven-burmeister)]

This only happens for me to messages which get marked as spam by bogofilter, i.e. piped through an external app and marked as spam.

The spam filtering, i.e. piping filter, is the last one in the list. All messages that do not reach that filter but are just filtered on List-ID etc. do not re-appear.

[Alexandre-bonneau (alexandre-bonneau)]

Duplicate mails no only happens when you go into a folder containing unread mails.
For instance if you have 2 unread mail in folder 'mymails', when you select that folder the unread mails are duplicated 'live' as ~32 unread mails.

I do use bogofilter, but those duplicated mails aren't marked as spam.

[rabauke (sven-burmeister)]

But they are piped through bogofilter before they get moved?

Without logs and debug output no dev will be able to understand what the issue is. Not mentioning the need to reproduce it in order to know what to fix.

So those that experience this should get that info into the report.

Please use 4.9.1 for testing since filtering did receive a lot of fixes and findings with 4.8 are a lot less likely to help debugging and hence fixing.

You can get debugging by enabling it for akonadi etc. in kdebugdialog and restarting akonadi in a konsole with akonadictl restart. akonadiconsole does also offer possibilities to get debug info.

In KMail > Tools/Extra you can find a possibility to log filtering.

Starting akonaditray might also be handy.

For imap see: http://techbase.kde.org/Projects/PIM/Akonadi/Debug_IMAP

[Alexandre-bonneau (alexandre-bonneau)]

I used the filter logging tools as pointed, and when opening a folder containing unread mails, they get duplicated, but nothing shows in the log.

Which components should we activate in kdebugdialog, kmail and all akonadi* entries ?

[Alexandre-bonneau (alexandre-bonneau)]

This bug is still affecting 4.9.1 by the way (4:4.9.1-0ubuntu1~ubuntu12.04.1~ppa2).

[rabauke (sven-burmeister)]

(In reply to comment #35)
> I used the filter logging tools as pointed, and when opening a folder
> containing unread mails, they get duplicated, but nothing shows in the log.

AFAIK filtering is not applied to unread emails but just to new emails, i.e. "just arrived" in the inbox. However, duplication taking place when switching folders would hint towards filtering not being the cause of the duplication.

> Which components should we activate in kdebugdialog, kmail and all akonadi*
> entries ?

Yes. To see that output issue akonadictl restart from a konsole and/or watch the general log file with tail -f ~/.xsession-errors

[Alexandre-bonneau (alexandre-bonneau)]

I activated all akonadi* and kmail occurence in kdebugdialog, then restarted akonadi in a console.

Browsing like always in folders containing unread mails, I sometimes see :
"QStringList Akonadi::NepomukSearch::search(const QString&) Calling blockingQuery() failed!
QStringList Akonadi::NepomukSearch::search(const QString&) Calling blockingQuery() failed!", but it doesn't necessarily duplicates unread mails.
That doesn't help I guess, but I can't really see any special debug lines tied specifically to the problem when it appears in kmail..

I do get the usual " AkonadiAgentServer(14273) MaildirResource::itemRemoved: Item removed 379717 in collection : 625" repeated over and over when I use the function to delete duplicated mails.

~/.xsession-errors on the other hand is much more verbose. When entering a folder and seeing the duplicate bug happening, I get this :
kmail2(15566) MessageList::StorageModel::StorageModel: Using model: Akonadi::EntityTreeModel
kmail2(15566) Nepomuk::ResourceWatcher::start: Failed to connect to watch service "The name org.kde.nepomuk.DataManagement was not provided by any .service files"
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) MessageList::Core::View::slotSelectionChanged: View message selected [ " Debian Project News - September 17th, 2012" ]
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done
kmail2(15566) KMReaderWin::setMessage: void KMReaderWin::setMessage(const Akonadi::Item&, MessageViewer::Viewer::UpdateMode) QSplitter(0x1c79610, name = "splitter2")
kmail2(15566) makeShowAuditLogLink: not showing link (not implemented)
kmail2(15566) KEmoticonsTheme::tokenize: Broken HTML entity, trying to recover.
...repeating 50 times...
kmail2(15566) KEmoticonsTheme::tokenize: Broken HTML entity, trying to recover.
kmail2(15566) MailCommon::SendMdnHandler::Private::handleMessages: AKONADI PORT: Disabled code in void MailCommon::SendMdnHandler::Private::handleMessages()
kmail2(15566) makeShowAuditLogLink: not showing link (not implemented)
kmail2(15566) KEmoticonsTheme::tokenize: Broken HTML entity, trying to recover.
kmail2(15566) KEmoticonsTheme::tokenize: Broken HTML entity, trying to recover.
...repeating 50 times...
kmail2(15566) KEmoticonsTheme::tokenize: Broken HTML entity, trying to recover.
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a mail check.
kmail2(15566) KMKernel::slotProgressItemCompletedOrCanceled: Last resource finished syncing, mail check done"

sometimes I see that :
"kmail2(15566) MessageList::StorageModel::StorageModel: Using model: Akonadi::EntityTreeModel
kmail2(15566) Nepomuk::ResourceWatcher::start: Failed to connect to watch service "The name org.kde.nepomuk.DataManagement was not provided by any .service files"
kmail2(15566) KMKernel::instanceStatusChanged: A Resource started to synchronize, starting a...

[J-diel (j-diel)]

OS: Linux
KMail Version: 4.8.5

Dear,
I'd like to rephrase the problem:
"Inbox" is displayed empty.
When I select the inbox folder, I see old mails re-appearing for a fraction of a second, but my filtering rules move the mails to their destination folders, where they appear again as "unread" and create duplicates. The "Inbox" looks empty again.
<F5> (I think it is "refresh" in English) while in the inbox triggers the same behavior.
My theory: This wrong behavior has nothing to do with the filtering rules, they work just fine.
The problem is, that old emails, which have been moved away from "inbox" already and seem to be away in the frontend, are still lurking in the dark and magically reappear when the inbox is refreshed.
Basic problem acc. to this theory: When a mail is moved, it looks moved in the front-end, but it is not really moved.
For me, this problem is really severe, because it makes it impossible to identify really new mails and creates an enormous # of duplicates (and "delete duplicates" sometimes works, sometimes not or it takes forever, and can't be done recursively).

[J-diel (j-diel)]

I could verify my theory:
 - switched off all filters
 - <F5> in inbox
 - about 50 old mails reappeared w/o being moved by filters
 - really deleted these mails (<shift> + <del>)
 - <F5> in inbox
 - no old mails showed up again (q.e.d)

[Andras Mantia (amantia)]

Can anybody reproduce the problem with KDE 4.9.2 (and preferably Akonadi 1.8.1)?
If yes, what kind of account does the filter run on (maildir, mixedmaildir, mbox, imap - if imap, online or disconnected)?
Does the mails come up if you delete simply with "Del" and refresh after?

[J-diel (j-diel)]

Since my previous entry, I changed to these versions of KMail and Akonadi = 4.9.2-356.1-x86_64
Account = maildir
del and refresh now do NOT make the mail reappear anymore, but:
1. Move a mail manually from the inbox to to a subdir
2. Mail is not visible anymore in inbox, but visible in the subdir
3. <F5> Refresh: mail is visible in Inbox AND subdir, i.e., duplicate

[Andras Mantia (amantia)]

Is the subdir a subirectory of inbox or a different subdirectory? Just to double check, is it maildir or mixedmaildir account?
Can you check that after the move (but before refresh), is the mail still physically on the disk in the inbox folder? You can check the location in the account settings, and locate the mail file e.g with a search in content on the file system.

[J-diel (j-diel)]

It is 3 levels below inbox, like:
/inbox/L1/L2/targetOfMove
in the properties of the target folder, it says "maildir"
before move: /home/juergen/.local/share/local-mail/Posteingang/cur/1350028982.R502.DIY-TUX:2,S
("Posteingang" = German for "Inbox")
after move: Still in the same directory.

[Anders Lund (anders-alweb)]

Created attachment 74537
imap log, a duplicate message is fetched

Since that other bug does not fit the problem I experience, I attach my logs here. More to follow.

[Anders Lund (anders-alweb)]

Created attachment 74538
filter log, a duplicated message is fetched

This should be the counterpart to the imap log attached above.

[Andras Mantia (amantia)]

 Which one gets duplicated, the spam (Donación de la Hermana Alice Jahab)
or the mail sent to @kde.org ([Bug 283548] Only first filter in chain works for incoming messages)?
I see the second (the kde.org mail) is deleted from inbox, so i guess that's not the problematic one.

The spam is more interesting. Can you paste the two filters that go through it from the akonadi_mailfiter_agentrc?
These ones:
[11:48:19] Evaluerer filterregler: (match all of the following) "<size>" <greater> "0"

[11:48:19] 1 = "<size>" <greater> "0" ( 14414 )

[11:48:19] Filterregler er blevet matchet.

[11:48:19] Applying filter action: Pipe Through "bogofilter -p -e"

[11:48:28] Evaluerer filterregler: (match any of the following) "X-Bogosity" <regexp> "(yes)|(spam\b)"

[11:48:28] 1 = "X-Bogosity" <regexp> "(yes)|(spam\b)" (Spam, tests=bogofilter, spamicity=1.000000, version=1.2.2)

What should happen here is:
- mail is moved from IMAP inbox to local folder
- mail is marked as spam and read (in the local folder)

From the IMAP log though what happens is:
- the mail is appended again to the inbox immediately after it is fetched and marked as spam and read (\Seen $JUNK)
- the original mail is deleted (two times!)
Seems like the filter actions are executed in wrong order, resulting in code duplication.

[Marcelo (mmtsales)]

I'm seeing this in 4.9.2 (Kubuntu 12.04 x86_64)
Several messages get duplicated when filtered, and very often the duplicate one has only the header, no body. When this error happens, only one duplicate is created for each message affected.
The duplicated message (the one without a body) is sometimes left in inbox and sometimes moved automatically to the filter destination folder along with the original message. Sometimes the duplicate stays in the inbox, but as soon as I click the inbox the message is moved automatically to the filter destination folder. Sometimes, it is moved only if I trigger the filters manually with CTRL+J.
Almost always I can delete both the original and the duplicate by pressing DEL. Sometimes, however, one of them can't be deleted. When I press DEL the selection goes to the next message, but the message I tried to delete remains where it was. I can then repeat this several times with that message and it will not go away. If I close KMail and reopen it, the problematic message is gone from the folder where it was.

[Marcelo (mmtsales)]

I forgot to mention, I get e-mails only from POP3 accounts (no IMAP) and they are locally stored in a maildir storage.

[Marcelo (mmtsales)]

I'm seeing this in 4.9.2 (Kubuntu 12.04 x86_64)
Several messages get duplicated when filtered, and very often the duplicate one has only the header, no body. When this error happens, only one duplicate is created for each message affected.
The duplicated message (the one without a body) is sometimes left in inbox and sometimes moved automatically to the filter destination folder along with the original message. Sometimes the duplicate stays in the inbox, but as soon as I click the inbox the message is moved automatically to the filter destination folder. Sometimes, it is moved only if I trigger the filters manually with CTRL+J.
Almost always I can delete both the original and the duplicate by pressing DEL. Sometimes, however, one of them can't be deleted. When I press DEL the selection goes to the next message, but the message I tried to delete remains where it was. I can then repeat this several times with that message and it will not go away. If I close KMail and reopen it, the problematic message is gone from the folder where it was.
I get my e-mails only from POP3 accounts (no IMAP) and they are locally stored in a maildir storage.

[Alexandre-bonneau (alexandre-bonneau)]

For info, I checked if the messages were really physically duplicated in the folders, or only in akonadi cache ; they are duplicated each 17 times physically.
Using maildir, each mail has a distinct name like '1350381035.R668.linux', '1350352842.R783.linux:2,S' or '1349859050.R458.linux:2,FS'.

This problem still means NOGO for use in a production environment.

[Andras Mantia (amantia)]

FYI, I have a tentative patch for this, but need to clean up and backport to 4.9 branch. I will post here once it is done, so you can test it.

[Andras Mantia (amantia)]

Git commit 37515fa641381b1357f26b0372f69bf9300dc10f by Andras Mantia.
Committed on 18/10/2012 at 22:53.
Pushed by amantia into branch 'KDE/4.9'.

Backport of the filter pipeline refactoring fix, that should fix the duplication of mails when it goes through several filter that move and modify the mail.
Related: bug 305073
FIXED-IN: 4.9.3

M +29 -41 mailfilteragent/filtermanager.cpp

http://commits.kde.org/kdepim/37515fa641381b1357f26b0372f69bf9300dc10f

[Alexandre-bonneau (alexandre-bonneau)]

Using Kmail 4:4.9.3-0ubuntu0.1~ppa1, I indeed see close to no more duplication !
Great job !!

I did saw 2 or 3 duplications while browsing previously unread mails (mails that were fetched before using 4.9.3), but so far, so good.

I'll report back here in a week for my last statement on this annoying bug.

[Christian Trippe (christiandehne)]

I still get duplicated messages with KDE 4.9.3 when downloading mails from a pop3 account which then get directly moved to a subfolder via a filter. Both the original and the duplicate message are displayed in the destiantion folder of the filter and you can "remove" the duplicate by right click on the folder and choosing "refresh folder".

[Mark Fraser (launchpad-mfraz)]

I'm also getting duplicated messages with KDE 4.9.3 and Kubuntu 12.10. I have turned logging on to try to help with the bug reporting, but I seem to be getting fewer duplicates with it turned on.

[Andras Mantia (amantia)]

Christian: what kind of accounts are involved (source/destination). Can you post the filter in question from the filter config file (akonadi_mailfilter_agentrc) and the log? Which duplicate disappears on the refresh (original or dest folder)?

[Christian Trippe (christiandehne)]

The account is a pop3 account

christian@asterix:~> cat .kde4/share/config/akonadi_pop3_resource_2rc
[General]
host=pop.gmx.net
intervalCheckEnabled=true
leaveOnServerCount=0
leaveOnServerDays=0
leaveOnServerSize=0
<email address hidden>
storePassword=true
targetCollection=35
useTLS=true

My local collection is a maildir-resource

The filter is
[Filter #10]
Applicability=1
AutomaticName=false
ConfigureShortcut=false
ConfigureToolbar=false
Enabled=true
Icon=system-run
StopProcessingHere=true
ToolbarName=<List-Post>:<mailto:<email address hidden>>
action-args-0=44
action-name-0=transfer
actions=1
apply-on=check-mail,manual-filtering
contentsA=<mailto:<email address hidden>>
fieldA=List-Post
funcA=contains
identifier=A6fHojqTOAJOHYm7
name=<List-Post>:<mailto:<email address hidden>>
operator=and
rules=1

Both files are in the destination folder. But there is only one mail under .local/share/.local-mail.directory/some/sub/directory but two ar under ./local/share/akonadi/file_db_data/

As soon as the last folder gets cleaned up. The second mail which kmail displays vanishes.

Regarding the logs: I am not sure how to obtain them.

[Alexandre-bonneau (alexandre-bonneau)]

After one week of intense testing, I see no more "17-duplicates" happening, which make Kmail usable one again, nice work there !

I did see once Kmail duplicate 2 mails with 1 clone each. The cloned mails shows no content, only headers, and unfortunately I realized that after deleting the good mails.
I use IMAP + Maildir.

As I said, I saw the behavior only one in one week, So...so far so good.

[Travisgevans (travisgevans)]

I still have the problem described in Comment 54 in KDE 4.9.3. Maildir, POP3. I have three messages that keep appearing in inbox every time I try to move, mark as read, or delete them. I also get a ton of “Maildir '' for collection '' is invalid” when I change folders. Anything I can do to try to figure out what's going on?

[Andras Mantia (amantia)]

Travis: your problem might be the one you describe in bug 288020, unrelated to the filtering.

[Alexandre-bonneau (alexandre-bonneau)]

To be more precise, even after this fix, you can still see the "17-duplicates" bug happens, not on mails that came since you started to use 4.9.3, but on previously received mails ; I had to click on every single folder of my mail folder tree to see some duplications, and remove duplicates manually.

I guess now I checked all my folders, and regarding how the fix from András fixed things for me, I won't see anymore duplication, hopefully.

[Holy-t (holy-t)]

Are there any chances this will be backported to 4.8 and hence appear in the default packages for Kubuntu 12.04? Or should I upgrade to 4.9?

[Frédéric Meynadier (frederic-meynadier-b)]

Bug still present in 4.9.4 (4.9.4-0ubuntu0.1) :

I have been using Kmail 4.9.4 for a week (looking for an alternative to thunderbird). Yesterday I set up filters (both custom filters and spamassassin filters that where automatically enabled by Kmail when I checked the "spam filter" option). It worked fine until, this morning, some mailing-list message I received was continuously duplicated as long as I had been connected to my mail account (through IMAP).

Once I had figured out that Kmail was responsible for this, I found this thread and deactivated the filters : the duplication stopped (after ~300 mails where added to my mailbox, making it quite difficult to regain control until I found how to stop the flux).

Kmail has very nice features and I was quite enjoying it until this problem arose, but this was *scary* !

[Marc Collin (collinm)]

problem exist in 4.10.2

[Alexandre-bonneau (alexandre-bonneau)]

Using 4.10.2 since its release, I haven't encountered this bug so far.

[Peter (jynyl)]

Similar symptoms in kmail 4.10.2 in Kubuntu 13.04.

This is a fresh install of Kubuntu 13.04 64 bit. The duplicated emails are from a pop account, and filtered to different folders.
Similar problem occurred with earlier versions of kmail, but it only ever made 1 copy. Sometimes, Kmail 4.10.2 has made 3 or 4 copies of an email. Many emails are handled correctly and are not duplicated.

[Hussam-v (hussam-v)]

problem still in 4.10.3
akonadictl stop
rm -rf ~/.local/share/akonadi
akonadictl start

remvoes the duplicates.

[Hussam-v (hussam-v)]

same email being filtered twice

---------------------
[18:50:24] Begin filtering on message "[Bug 627699] Port GTK2 to GTK3" from "<email address hidden>" at "Monday 08 July 2013 15:47" :
[18:50:24] Evaluating filter rules: (match any of the following) "From" <contains>
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "<email address hidden>"
[18:50:24] 0 = "From" <contains> "<email address hidden>" (<email address hidden>)
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "<email address hidden>"
[18:50:24] 0 = "From" <contains> "<email address hidden>" (<email address hidden>)
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "<email address hidden>"
[18:50:24] 0 = "From" <contains> "<email address hidden>" (<email address hidden>)
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "<email address hidden>"
[18:50:24] 0 = "From" <contains> "<email address hidden>" (<email address hidden>)
[18:50:24] Evaluating filter rules: (match any of the following) "Sender" <contains> "<email address hidden>" "Reply-To" <contains> "<email address hidden>"
[18:50:24] 0 = "Sender" <contains> "<email address hidden>" ()
[18:50:24] 0 = "Reply-To" <contains> "<email address hidden>" ()
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "<email address hidden>"
[18:50:24] 0 = "From" <contains> "<email address hidden>" (<email address hidden>)
[18:50:24] Evaluating filter rules: (match all of the following) "From" <contains> "<email address hidden>"
[18:50:24] 1 = "From" <contains> "<email address hidden>" (<email address hidden>)
[18:50:24] Filter rules have matched.
[18:50:24] Applying filter action: Move Into Folder "Local Folders/inbox/Bugs/Mozilla"
------------------------------
[18:50:25] Begin filtering on message "[Bug 627699] Port GTK2 to GTK3" from "<email address hidden>" at "Monday 08 July 2013 15:47" :
[18:50:25] Evaluating filter rules: (match all of the following) "From" <contains> "<email address hidden>"
[18:50:25] 0 = "From" <contains> "<email address hidden>" (<email address hidden>)
[18:50:25] Evaluating filter rules: (match all of the following) "From" <contains> "<email address hidden>"
[18:50:25] 0 = "From" <contains> "<email address hidden>" (<email address hidden>)
[18:50:25] Evaluating filter rules: (match all of the following) "From" <contains> "<email address hidden>"
[18:50:25] 0 = "From" <contains> "<email address hidden>" (<email address hidden>)
[18:50:25] Evaluating filter rules: (match all of the following) "From" <contains> "<email address hidden>"
[18:50:25] 0 = "From" <contains> "<email address hidden>" (<email address hidden>)
[18:50:25] Evaluating filter rules: (match any of the following) "Sender" <contains> "<email address hidden>" "Reply-To" <contains> "<email address hidden>"
[18:50:25] 0 = "Sender" <contains> "<email address hidden>" ()
[18:50:25] 0 = "Reply-To" <contains> "<email address hidden>" ()
[18:50:25] Evaluating f...

[Hussam-v (hussam-v)]

I will open a new bug. seems this is a different issue. sorry for the spam.

[Kde-6 (kde-6)]

4.10.5 I experience the problem.

[Jaanus-r (jaanus-r)]

Still not fixed. Account which has filters will duplicate messages which are not matched by any filters ad infinitum. Which is extremely annoying.

When will it be fixed?

$ rpm -q kdepim
kdepim-4.11.5-1.fc20.x86_64

[Frederic-coiffier-k (frederic-coiffier-k)]

Still occurs in KDE 4.12.1 (Gentoo)...

I've just deleted my ~/.local/share/akonadi as mentionned in a comment above.
I'll see in the next days if it changes something...

I have also delayed the Kmail startup as the problem occurs often when the whole KDE session is restored (in this case, the net connection can be overloaded and emails download can be slow...).

[Jaanus-r (jaanus-r)]

Still broken in kdepim-4.12.2-1.fc20.x86_64.

Is this issue being worked on?

[Kde-bugs-3 (kde-bugs-3)]

I also still see this bug with KDE 4.12.1 - reopening...

[Frederic-coiffier-k (frederic-coiffier-k)]

I confirm too (kdepim 4.12.2 and akonadiserver 1.11.0). I can't find a working solution : deleting ~/.local/share/akonadi, removing bogofilter filters, delaying KMail start. Nothing can solve the problem. And the problem is mainly due to a slow harddisk (due to a highload) or a slow network connection.

[Dvratil (dvratil)]

If you can reproduce this manually on one or two emails, please start Akonadi Console, enable the "Debugger", "Job Tracker" and "Notification Monitor" tabs and try to reproduce the issue.

Then provide output from the "All" tab in "Debuger" tab and screenshots from the other two tabs (please expand all subtrees). Sorry, we don't have text export yet :-)

[Manfred-1980 (manfred-1980)]

Created attachment 85825
Akonadi Console Notification Monitor

[Manfred-1980 (manfred-1980)]

the debugging log was actually quite fast 130MB big and had some emails content. I dont feel like uploading it. But, the email content in the debuglog was exact the one, which was appearing 7 times (now) as new email.

[Manfred-1980 (manfred-1980)]

I first deleted all filters, only left Spam and virus filters. Before I had a filter sorting some emails by sender into a subfolder of the SENT folder. Lets call the subfolder 123.

After manually moving emails from the SENT folder to the subfolder 123 they reappear soon in the original SENT folder, now doubled, one in the subfolder 123, one in the SENT folder. The subfolder 123 was created a longer time ago.

[Manfred-1980 (manfred-1980)]

and as other people described, it's an ongoing duplication - sometimes. And it not every email I manually put into that subfolder 123.

[Hussam-v (hussam-v)]

It seems like I hit this with kmail 4.13rc1 when I manually check for mail when there are emails being currently downloaded and filtered.

[Νίκος Αλεξανδρής (nikos.alexandris)]

I think iI experience the same here under KDE Development Platform: 4.12.0, KMail: 4.12. Very annoying.

[Jwagner-5 (jwagner-5)]

The bug seems to be triggered by something that doesn't happen very often. I've been using kmail in kontact 4.11.5 for 8 days without problems and now just got hit by this bug.

While this report is not for the most recent version, the following observations may help the developers to reproduce the bug. Like other bug reporters above, I cannot attach akonida debug log files as they contain confidential information.

Setup:
 * fresh OpenSUSE 13.1 with default updates (kmail/kontact 4.11.5)
 * fresh home directory
 * pop3 account connected to inbox
 * approximately 80 destination folders under "Local Folders"
 * no automatic filters defined
 * manually moving e-mails to destination folders
 * often using the "Search" input field over the message list and selecting multiple messages (shift + left click or ctrl + left click) to move multiple messages in one go

Observation:
 * in my case, the problem seems to be related to a specific local destination folder (it was newly created just before the bug was observed, let's call this folder "A") as the moved message doesn't re-appear as a duplicate in the inbox if I move it to a different destination folder (let's call it "B") and press F5
 * if I move the message from folder B to folder A, it is still duplicated in folder B (after pressing F5) but not in the inbox
 * if I create a new local folder "C", move the message from B to C and press F5, everything is normal
 * however, if I create "C2" under the A or the parent of A (which had been created just seconds before A) and move a message to it, duplication occurs

Workaround 1:
 * the "copy first, then delete with ctrl-del" trick worked for me

Workaround 2:
 * create a new folder "A-Parent-2" and subfolder "A2"
 * move message from A to A2
 * press F5 (no duplicates should re-appear in A)
 * select A2 and run "Folder - Remove Duplicate Messages"
 * delete empty folders A and A-Parent
 * rename A2 to A and A-Parent-2 to A-Parent

JJ

[Hussam-v (hussam-v)]

on kmail 4.13.2, this happens with every single filtered email.
pop3 account.
I end up stopping akonadi and clearing the database, starting akonadi, importing the filters again over a 100 times a day which is too time consuming.
Any way i can monitor some logs to tell what is happening?
This is a really bad bug. I hate to complain and rant (because complaining doesn't solve bugs) but it sounds that this isn't getting the attention it needs.
At least tell us please what we can do to help.
Thank you.

[Cgiboudeaux (cgiboudeaux)]

(In reply to comment #81)
> Any way i can monitor some logs to tell what is happening?

tools / Filter Log Viewer
Check "Log filter activities" and the relevant boxes below then run your filter.

Don't forget to uncheck "Log filter activities" once done

[Dandress2 (dandress2)]

On version 4.13.2 this problem began two days ago. There may have been Fedora updates precipitating the start.

Duplicate emails are disabled in the message list and can not be selected or deleted. If I stop Kontact/Kmail and start another email client, SeaMonkey in my case, the duplicate emails will not be present. If, and only if, I select the folder which contained duplicates in KMail, and then restart KMail, the duplicates will be gone.

[Hussam-v (hussam-v)]

(In reply to comment #83)
> On version 4.13.2 this problem began two days ago. There may have been
> Fedora updates precipitating the start.
>
> Duplicate emails are disabled in the message list and can not be selected or
> deleted. If I stop Kontact/Kmail and start another email client, SeaMonkey
> in my case, the duplicate emails will not be present. If, and only if, I
> select the folder which contained duplicates in KMail, and then restart
> KMail, the duplicates will be gone.

They won't show up in another mail client because the duplicates are in akonadi database.

[Dandress2 (dandress2)]

(In reply to comment #84)
> (In reply to comment #83)
>
> They won't show up in another mail client because the duplicates are in
> akonadi database.

I hadn't considered that, thank you. Yet, restarting KMail does not fix the problem, running akonadictl vacuum and fsck does not fix it either. But, viewing mail from another client does, or at least seems to affect the problem.

[0843-j (0843-j)]

New to KDE. Running Fedora 20, KDE 4.13.2. Created about 20 email identities, with about the same number of filters to simply move messages into appropriate folders. Filter processing is duplicating email messages. If I disable filters, problem goes away. Example: With filters enabled my inbox contains three copies of the KDE Bug Tracking Account Creation email (I just signed up to provide this input). Each time I click Check Mail, additional email messages are duplicated for new email messages.

Am surprised and disappointed such a fundamental, common feature does not work. Must seek another email client solution as a result.

[Schuschq (schuschq)]

had the same problem,

after migrating a really big mailbox (lots of old mails, mostly small mails, using 1,9 GB now on drive) from pine some two years ago, i used another program on opensuse 12.x (the standard one, forgot the name) and transferred all the mailbox folders to it. After setting up a new linux box with opensuse 13.1 I finally transferred all the mail to K-Mail - which all worked without problems. But since introducing new filters in K-Mail everytime i changed to the incoming mail folder (Posteingang) now, I had a set of spam mails which always reappeared ... they are then immediately filtered in a spam folder ... in this folder I had dozens of duplicates after some weeks. The duplicates all "belonged" to one of the filters.

here: Opensuse 13.1, K-Mail/KDE 4.11.5

what I did to get rid of this duplicates

- i changed the folder for incoming mail to anotherone (e.g. _incoming)
- i moved all the mails from the incoming folder to _incoming
- quitted K-Mail

- found still mail in
  /home/schorsch/.local/share/local-mail/inbox/new/
  (all from this spamming shopping-de.cc mailer)

- deleted these mails manually

- opened K-Mail again, switched incoming mail folder back to original folder
- quitted K-Mail, opened it again, moved mail back to original folder
- done

no more duplicates :-) ... perhaps this helps?

[Schuschq (schuschq)]

(In reply to schuschq from comment #87)
> what I did to get rid of this duplicates

> no more duplicates :-) ... perhaps this helps?

but the next new message hitting the filter leads to new duplicates ... so still not solved (at least the old duplicates are no longer replicated)

[B-misc-t (b-misc-t)]

(In reply to Joachim Wagner from comment #80)
> The bug seems to be triggered by something that doesn't happen very often.
> I've been using kmail in kontact 4.11.5 for 8 days without problems and now
> just got hit by this bug.
>
> While this report is not for the most recent version, the following
> observations may help the developers to reproduce the bug. Like other bug
> reporters above, I cannot attach akonida debug log files as they contain
> confidential information.
>
> Setup:
> * fresh OpenSUSE 13.1 with default updates (kmail/kontact 4.11.5)
> * fresh home directory
> * pop3 account connected to inbox
> * approximately 80 destination folders under "Local Folders"
> * no automatic filters defined
> * manually moving e-mails to destination folders
> * often using the "Search" input field over the message list and selecting
> multiple messages (shift + left click or ctrl + left click) to move multiple
> messages in one go
>
> Observation:
> * in my case, the problem seems to be related to a specific local
> destination folder (it was newly created just before the bug was observed,
> let's call this folder "A") as the moved message doesn't re-appear as a
> duplicate in the inbox if I move it to a different destination folder (let's
> call it "B") and press F5
> * if I move the message from folder B to folder A, it is still duplicated
> in folder B (after pressing F5) but not in the inbox
> * if I create a new local folder "C", move the message from B to C and
> press F5, everything is normal
> * however, if I create "C2" under the A or the parent of A (which had been
> created just seconds before A) and move a message to it, duplication occurs
>
> Workaround 1:
> * the "copy first, then delete with ctrl-del" trick worked for me
>
> Workaround 2:
> * create a new folder "A-Parent-2" and subfolder "A2"
> * move message from A to A2
> * press F5 (no duplicates should re-appear in A)
> * select A2 and run "Folder - Remove Duplicate Messages"
> * delete empty folders A and A-Parent
> * rename A2 to A and A-Parent-2 to A-Parent
>
> JJ

This might have the same reason than something I observed just this weekend:
I added a new folder to my local folders and it took several seconds until it was shown. Afterwards I manually moved messages to this new folder. The showed up correctly in KMail, but no in the corresponding folder on my disk (.local/share/local-mail/...) so they seem to stay in the akonadi cache forever. Just a second after trying to move them the reappear in the original folder. The problem didn't show up with another folder I created. So it looks like the main problem being a folder not being created correctly.

[Hussam-v (hussam-v)]

On Monday 18 August 2014 12:08:03 you wrote:
> https://bugs.kde.org/show_bug.cgi?id=283682
>
> mau <email address hidden> changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> CC| |<email address hidden>
>
> --- Comment #89 from mau <email address hidden> ---
> (In reply to Joachim Wagner from comment #80)
>
> > The bug seems to be triggered by something that doesn't happen very often.
> > I've been using kmail in kontact 4.11.5 for 8 days without problems and
> > now
> > just got hit by this bug.
> >
> > While this report is not for the most recent version, the following
> > observations may help the developers to reproduce the bug. Like other bug
> > reporters above, I cannot attach akonida debug log files as they contain
> > confidential information.
> >
> > Setup:
> > * fresh OpenSUSE 13.1 with default updates (kmail/kontact 4.11.5)
> > * fresh home directory
> > * pop3 account connected to inbox
> > * approximately 80 destination folders under "Local Folders"
> > * no automatic filters defined
> > * manually moving e-mails to destination folders
> > * often using the "Search" input field over the message list and
> > selecting
> >
> > multiple messages (shift + left click or ctrl + left click) to move
> > multiple messages in one go
> >
> > Observation:
> > * in my case, the problem seems to be related to a specific local
> >
> > destination folder (it was newly created just before the bug was observed,
> > let's call this folder "A") as the moved message doesn't re-appear as a
> > duplicate in the inbox if I move it to a different destination folder
> > (let's call it "B") and press F5
> >
> > * if I move the message from folder B to folder A, it is still duplicated
> >
> > in folder B (after pressing F5) but not in the inbox
> >
> > * if I create a new local folder "C", move the message from B to C and
> >
> > press F5, everything is normal
> >
> > * however, if I create "C2" under the A or the parent of A (which had
> > been
> >
> > created just seconds before A) and move a message to it, duplication
> > occurs
> >
> > Workaround 1:
> > * the "copy first, then delete with ctrl-del" trick worked for me
> >
> > Workaround 2:
> > * create a new folder "A-Parent-2" and subfolder "A2"
> > * move message from A to A2
> > * press F5 (no duplicates should re-appear in A)
> > * select A2 and run "Folder - Remove Duplicate Messages"
> > * delete empty folders A and A-Parent
> > * rename A2 to A and A-Parent-2 to A-Parent
> >
> > JJ
>
> This might have the same reason than something I observed just this weekend:
> I added a new folder to my local folders and it took several seconds until
> it was shown. Afterwards I manually moved messages to this new folder. The
> showed up correctly in KMail, but no in the corresponding folder on my disk
> (.local/share/local-mail/...) so they seem to stay in the akonadi cache
> forever. Just a second after trying to move them the reappear in the
> original folder. The problem didn't show up with another folder I created.
> So it looks like the main problem being a folder n...

[Ago-sylvain (ago-sylvain)]

Using Kmail version 4.14 on archlinux. I enabled bogofilter and I have the problem of multiple copies (thousands!) of the same mail when a new mail arrives. seems to be the same as this bug. If I disable bogofilter, the problem disappear.

I sent me an email from $<email address hidden> to $PROMAIL@univ_mail and logged everything. I waited to have multiple copies (5) before putting kmail offline. when I put it online again it will download one more copies (read state directly) and stop there (sorry no log).

Hope this can help

[Ago-sylvain (ago-sylvain)]

Created attachment 88424
akonadi log all

[Ago-sylvain (ago-sylvain)]

Created attachment 88425
akonadi log simple

[Ago-sylvain (ago-sylvain)]

Created attachment 88426
akonadi maildir resource

[Ago-sylvain (ago-sylvain)]

Created attachment 88427
akonadi mailfilter agent

[Ago-sylvain (ago-sylvain)]

Created attachment 88428
new mailnotifier agent

[Ago-sylvain (ago-sylvain)]

Created attachment 88429
mail filter kernel ETM

[Ago-sylvain (ago-sylvain)]

Created attachment 88430
kmail filter log applying the bogofilter pipe

[Dandress2 (dandress2)]

KMail 4.13.3
Fedora 20 (testing and unstable repos in use)

I had the problem where a spam filter would endlessly duplicate emails. I quickly removed the spam filter and things settled down. The problem returns when I delete or move an email. The email will move to the detination folder, only to shortly reappear, disabled and not selectable, in the original folder.

I have two email accounts configured, GMail and the my employer's (ms exchange server). The problem occurs almost exclussively with my employer's account. Emails duplicate in the GMail spam folder, curiously taking several minutes to show up instead of almost immediately.

I don't see akonadi being at fault. Running akonadictl fsck and vacuum show no errors, and don't affect the problem. Deleteing the Adkonadi database and starting over didn't help either. Yet, I can clear the duplicates by using any other mail client and synching each folder individually. To make a SWAG (stupid wild ass guess), I'd look for something in LDAP usage.

Sorry to report a problem and not offer help. Thanks.

[Dandress2 (dandress2)]

This bug is annoying...

I restarted Akonadi this morning and left the terminal window open. I then deleted several emails from one KMail folder and noticed this, repeated once for each deleted email, in the terminal window

akonadi_imap_resource_7(3524)/kdepimlibs (kimap) KIMAP::StoreJob::handleResponse: We asked for UID but the server didn't give it back, resultingFlags not stored.

After awhile I hacked around with Akonadi Console and eventually found that for the duplicated emails the table PIMITEMFLAGRELATIONS will have two records, one with the flag /SEEN and one with the flag /DELETED. I didn't look very hard, but it seems that pimitemflagrelations has only one record for emails that are not duplicated.

This SQL run from the DB Query tab of Akonadi Console seems to confirm that.

select * from pimitemflagrelation pif
INNER JOIN pimitemflagrelation pir ON pir.pimitem_id = pif.pimitem_id
where pif.flag_id = 6 and pir.flag_id = 2

-- where flag_id 6 == /DELETED and flag_id 2 == /SEEN

I admit that is poor SQL. I'm not a fan of MySQL, having decided long ago that its implementation of SQL is not compatible with my career goals.

It seems that message I saw in Akonadi's stdout is saying that the /SEEN record was not removed from pimitemflagrelations when the /DELETED record was added.

[Hussam-v (hussam-v)]

I found a way to clear the duplicates without clearing the whole akonadi
database.

Close kmail and open akonadiconsole.
In the browser tab, right click on the collection/folder where you have
duplicates.
Select the "Clear akonadi cache" context menu option.
Now open kmail and it will rescan that folder and hopefully the duplicates
will be gone.

[Tobias-leupold (tobias-leupold)]

Does not work for me. Even if so, this does not solve the problem, but only handles it … because we still have lots and lots of duplicate mails generated by filtering :-(

[Tobias-leupold (tobias-leupold)]

Well, I think the fact that clearing the akonadi cache is not removing the duplicate mails for me is mostly due to the fact that mail filtering actually duplicates mails physically.

When I get duplicated mails, I actually have two files with different names but the very identical content in my target maildir, created at the same time.

[Hussam-v (hussam-v)]

I don't think they are being duplicated physically. you might need to try once and make sure everything using akonadi is closed first before purging tables/database.
You will also need to clear the baloo database too to avoid double indexing.
But this is all just messy. I wonder why no one is investigating this thoroughly...

[Tobias-leupold (tobias-leupold)]

Honestly, I really do get the duplicate emails as separate files in the actual physical maildir.

[Hussam-v (hussam-v)]

Then there are two issues here ;(

[Tobias-leupold (tobias-leupold)]

(In reply to Hussam Al-Tayeb from comment #104)
> But this is all just messy. I wonder why no one is investigating this
> thoroughly...
I'm not completely sure if the KMail guys really use KMail themselves :-P

Even if we have two causes for the problem, it's still the right place here as the bug's name is "KMail duplicates filtered messages" – and that's what happens, no matter if it's caused by some Akonadi/Baloo/Whatever black hole or something different.

[Andras Mantia (amantia)]

If you look at the history, you see we care about it and use it. Unfortunately I have no real time for KMail these days, and for me it works. Both for POP3 and IMAP accounts, I have no mail duplication with filters, so it is really hard to do anything without spending a lot of time on it,

[Tobias-leupold (tobias-leupold)]

No harm meant :-) But I think there's a general filtering problem since KMail uses Akonadi. I have e. g. been using bogofilter since ever to filter my mails, and since the last few versions, some of my mails are piped through it, some are not.

I really don't want to start a discussion, as that has been done 1000 times elsewhere and this is not the right place to do so. But KMail before Akonadi has simply been rock solid and by far more reliable. These days, it's really hard (at least for me) to still stick to KMail with all the problems, esp. with this filtering issue which is really a PITA.

The problem is that there are so many components involved in KMail that it's almost impossible to describe a problem with a reproducible scenario. Sometimes it all works, most of the time it doesn't. And I have no idea how to debug it. And all you get is "for me it works" ...

[Hussam-v (hussam-v)]

multiprocess systems are vulnerable to race issues anyway.

[Hussam-v (hussam-v)]

To add even more complexity :( , I got an email now that was filtered into the KDE bugs folder and I saw duplicates for a second and it autocleared the duplicate.
Exactly how does akonadi/kmail filtering work? A copy is made, filtered and original copy deleted?

[Hussam-v (hussam-v)]

http://i.imgur.com/2svl8Hj.png
The extra copy cannot be clicked on and disappeared when I restarted kmail.

[Andras Mantia (amantia)]

Unless someone changed meantime, the way it works is:
- mail is delivered to Inbox
- the filtering code gets a notification of new mails
- it runs the filters on them and does what you have set up in the filter, like moves the mail

[Tobias-leupold (tobias-leupold)]

Perhaps, this helps?

I have two maildir resources. The "normal" ~/.local/share/local-mail and an additional ~/.local/share/tmp-mail, of which I don't make backups. All e. g. mailing list mails, automatically generated status updates of my homepage etc. are filtered to that place.

Practically each and every mail filtered to the tmp-mail resource appears twice in the destination folder, alway with a physically existing duplicate file with the very same content and creation date, but a completely different file name.

Could it be that filtering mails to another maildir resource is part of the problem? Perhaps, not everybody creates multiple maildirs and thus does not always experience the filtering problem?

[Hussam-v (hussam-v)]

(In reply to Tobias Leupold from comment #114)
> Perhaps, this helps?
>
> I have two maildir resources. The "normal" ~/.local/share/local-mail and an
> additional ~/.local/share/tmp-mail, of which I don't make backups. All e. g.
> mailing list mails, automatically generated status updates of my homepage
> etc. are filtered to that place.
>
> Practically each and every mail filtered to the tmp-mail resource appears
> twice in the destination folder, alway with a physically existing duplicate
> file with the very same content and creation date, but a completely
> different file name.
>
> Could it be that filtering mails to another maildir resource is part of the
> problem? Perhaps, not everybody creates multiple maildirs and thus does not
> always experience the filtering problem?

Only one maildir here. ~/.local/share/.local-mail.directory/

(In reply to András Manţia from comment #113)
> Unless someone changed meantime, the way it works is:
> - mail is delivered to Inbox
> - the filtering code gets a notification of new mails
> - it runs the filters on them and does what you have set up in the filter,
> like moves the mail
Sure but the duplication in my case is in the database ~/.local/share/akonadi and not in ~/.local/share/.local-mail.directory/
Maybe akonadi is thinking the cached copy in ~/.local/share/akonadi/file_db_data/ is a second copy and then making an extra entry in the database?

[Tobias-leupold (tobias-leupold)]

I just tried to track it down (once again). I deleted all config files and directories that belong to kmail and akonadi and started with a "fresh" install. I downloaded all mails from my account via pop3 (~17,000) and created new filters.

Filtering all the mails manually was no problem. Not a single mail was duplicated. It sometimes took almost a minute until the mails were actually moved and written to the respective place on the disc, but it worked.

As soon as I got new mails from my account, and they were filtered as they arrived, most of them were duplicated again.

I'm very sure this is a timing problem. First, the mail only exists inside the (fast) akonadi cache and is processed by the filter, which puts it in the target directory. It's also written on the respective place on the disc there. Then, some (milli)seconds later, the original mail is written to the inbox (which probably happens in a thread started at the very same time the filter agent starts which does not know what the filter agent does). And then, we have the un-filtered mail again inside the inbox. The inbox notifies the filter agent that there's some new mail inside the inbox and it's moved from there a second time by the newly started filter agent.

I'll eat my hat if this is not the very cause of the duplicated filtered mails. All the existing already-downloaded mails were already written to the disc and did not live only inside the Akonadi cache, and thus, they were not duplicated on filtering. For the few not-duplicated filtered mails, it actually worked to write them to the disc before the filter agent did his work. That's it.

Does really not one single KMail developer see this problem each and every day?! I'd really like to help to fix this if anybody told me how …

[Tobias-leupold (tobias-leupold)]

*** Bug 316419 has been marked as a duplicate of this bug. ***

[Tobias-leupold (tobias-leupold)]

To investigate this further I tried the following: I added a filter executing a "/usr/bin/sleep 0.3" for each mail downloaded. I did not get any duplicated filtered mails anymore after doing so.

[monteleo (monteleone-c)]

In the IMAP setting, selecting "Automatically compact folders" removes grayed messages

[monteleo (monteleone-c)]

In the IMAP setting, selecting "Automatically compact folders" removes grayed messages for me (kubuntu 14.10)

[Andras Mantia (amantia)]

Tobias Leopold, you are probably right of being a timing issue. If you can reproduce it, and could work on the code, the chances of fixing it would be high(er). As I pointed out already, I use POP3 and IMAP with local filters on two machines and didn't see this problem for a long time, so best would be to fix on a machine where it actually manifests it.
 It was a long time ago when I last worked on Akonadi/KMail code, but here are some hints:
- add debug output (possibly with timestamps) in the filtering and the akonadi code. You might have to do that both in the akonadi server and the akonadi resources (filter agent and the base classes)
- the filter agent is in kdepim/mailfilteragent, the starting point could be mailfilteragent.cpp and there the itemAdded() method. This is called when a new mail appeares in a monitored folder (ie. the inbox)
- itemsReceiviedForFiltering() there is called when the filter agent retrieves the above item (so it can analyze that it needs to be filtered or not)
- interesting code could be also in Akonadi::AgentBase class (kdepimlibs repository) and the changeRecorder() part, but only if you can't figure out anything useful from the above
- mailfilteragent/filtermanager.cpp is the code that does the filtering. processContextItem() is the method that does the move/copy of the messages. Adding debug output here also makes sense to see when does this happen
- for the server itself you can enable debugging in the akonadiconsole app, although this will give you lots of data and will make things slow, so better make sure you don't get lot of mails while debugging is enabled. But it can show all the communication between the akonadi server and the clients and all the jobs running and their order.

[Tobias-leupold (tobias-leupold)]

András, I'll have a look at the sources, but I fear the KMail/Akonadi codebase is a bit too hard for me (I have not been into the C++/Qt business very long yet ;-). But I'll try.

Another thing to substantiate the suspicion that the duplication does not happen when the mails are filtered but when they are delivered:

Yesterday, I saw that the default setting for filtering when selecting "Filter incoming messages" is that all "Accounts" are used for filtering. This includes not only the POP3 accounts (which are actual ones), but also the local maildir "Accounts".

I selected only my POP3 account and deselected everything else. I also deactivated my "/usr/bin/sleep 0.3" hack filter. After that, I ended up with filtered mails being written to their respective target directory and a (physical!) copy of those very mails being written to my inbox.

So it's actually the case that the emails are simply delivered twice, one by the filter agent and one by the, (?) pop3 agent, and the duplicating really happens because the inbox notifies new mails and the filter agent is invoked again.

Perhaps, this is "only" a sequence problem and the filter agent is invoked too early?! Can anybody who's into the code base right now check that? Otherwise, I fear we have a design problem here :-(

[Tobias-leupold (tobias-leupold)]

Ps: Interestingly, if I only choose my local maildir as the account to filter incoming mails (and deactivate the sleep hack filter), nothing happens and the mails are simply delivered to the inbox. According to the mail filter protocol, the mail filtering starts when the new mail is added, but no filter is invoked.

[Montel-3 (montel-3)]

Indeed I confirm the duplicate mail.

[Georg Abenthung (georg-0)]

Same issue here with KMail version 4.14.2 running on Kubuntu 14.10.
Also using bogofilter on my incoming mails on my IMAP-Account. Please let me know, if you need further information.

When receiving a mail KMail duplicates it endlessly (>500 times). I hav to turn off Akonadi server, log in onto my webmail-account, delete the mails, to keep one I move one copy to a different IMAP-Folder and restart Akonadi. The kept mail appears in the folder I moved it to, but it stays greyed out in Kmail. Happens now every-time, when I get a mail into KMail.

[Andrew Goss (aegoss-g)]

Netrunner 14 64 bit
Kmail 4.13.3
This bug affects all emails downloaded on startup, and some occasionally on subsequent checks. Emails in the Local Folders inbox and those filtered to other boxesare all affected. I Don't use Akonadi mail boxes. POP3.
I once spotted a triplicate message, but it vanished immediately.
This is a new installation, with Kmail data migrated from a Netrunner 14 32 bit (upgraded via script). Messy business that migration.

[3-eric-x (3-eric-x)]

Issue present also in kmail 4.13.3 and kubuntu 14.04. Emails get duplicated when manually moved from the inbox by drag and drop to any directory or when using the "Move Message To" Menu entry.

[Christof-kde (christof-kde)]

Same bug here. Opensuse 13.2, Kmail 4.14.4 :-(

[Christian Hilberg (hilberg)]

Seeing the same issue on Debian for 4.14.1 (jessie) as well as 4.14.2 (sid).
For me, duplicates occur only if mail filtering does NOT move the mail to a folder different from INBOX (using IMAP), where I let the filtering happen.

Moreover, "Folder"->"Remove Duplicate Messages" does not work. Of multiple duplicates, two will remain (if the operation is started at all after selecting this action, which does not seem to reliably be the case for me). I've seen an open bug report for this somewhere but I cannot seem to find it now.

Maybe there is some issue here in that Kmail/Akonadi do not reliably detect what ought to be considered "the same mail object", and hence the duplication occurs after the filters did not move the mail to a different folder?

[Christian Hilberg (hilberg)]

Duplicate of this bug: https://bugs.kde.org/show_bug.cgi?id=95064
Possibly related: https://bugs.kde.org/show_bug.cgi?id=288040
Possibly related: https://bugs.kde.org/show_bug.cgi?id=333035

[Kde4 (kde4)]

The bug is present in KMail 4.14.6 using Kubuntu 15.04. I also see duplicated Inbox messages appear in Inbox for a second, before disappearing.

[Eike-sauer (eike-sauer)]

Got duplicated mails as well, Debian Jessie, KMail 4.14.2.
The mails are pop3'ed to the inbox and moved by a (single) filter to their destination folder.

[Hussam-v (hussam-v)]

After months away, I found out today this is still heavily happening in KF5 Kmail port.
Half of my emails are getting duplicates in the akonadi database.
Truely a classic bug :)

[Christian Hilberg (hilberg)]

Confirming this still happens with KF5 Kontact/Kmail 4.14.10 on today's debian/stretch (where it seems almost all KF5 packages have materialized).

[René (rkrell)]

The problem does still appear in Akonadi + KMail 15.08.1.

Many duplicate mails from IMAP. Have just automatically created filter by the Anti Spam Wizard (Bogofilter) activated.

[René (rkrell)]

Created attachment 94708
Filter <unknown> after removing all filters manually - in this state it works

Once more for KMail + Akonadi 15.08.1.

After removing all four filter auto-generated by the Anti Spam Wizard for Bogofilter the messages from IMAP are retrieved in a regular manner again, without any duplicates. Before that I saw a race in the progress bars at the bottom right-hand side between the mail filter agent and the IMAP agent, which doen't happen now, but there is a hanging progressbar at 0% for the Mail Filter Agent.

Opening the filter dialog again results in an entry I've never made: <unknown>. In this state the messages are retrieved regularly.

[Hussam-v (hussam-v)]

(In reply to René Krell from comment #135)
> Created attachment 94708 [details]
> Filter <unknown> after removing all filters manually - in this state it works
>
> Once more for KMail + Akonadi 15.08.1.
>
> After removing all four filter auto-generated by the Anti Spam Wizard for
> Bogofilter the messages from IMAP are retrieved in a regular manner again,
> without any duplicates. Before that I saw a race in the progress bars at the
> bottom right-hand side between the mail filter agent and the IMAP agent,
> which doen't happen now, but there is a hanging progressbar at 0% for the
> Mail Filter Agent.
>
> Opening the filter dialog again results in an entry I've never made:
> <unknown>. In this state the messages are retrieved regularly.

<unknown> is the default empty filter which is there before you add yours. It doesn't do anything. Since this is likely some sort of race issue with filters, it's not going to appear when there are no filters (only the <unknown> one).

Maybe junk filtering and virus scans should not be exposed as filters but instead some internal plugins (with configuration options) that hook in as filters.

[René (rkrell)]

(In reply to Hussam Al-Tayeb from comment #136)
>
> <unknown> is the default empty filter which is there before you add yours.
> It doesn't do anything. Since this is likely some sort of race issue with
> filters, it's not going to appear when there are no filters (only the
> <unknown> one).
>
> Maybe junk filtering and virus scans should not be exposed as filters but
> instead some internal plugins (with configuration options) that hook in as
> filters.

To have an empty <unknown> filter without any setting makes no sense to me. I would put an empty list there. But this is just a view.

Worse is that filtering and mail fetcher resources actually have synchronization problems about four years (!) according to the creation time of this report.

I can provide logs or any information I'm able to gather from my system for analyzing. I'm not able to solve this.

[René (rkrell)]

Once more for KMail + Akonadi 15.08.1 regarding the hanging Mail Filter Agent progressbar at 0%: The progressbar hanging forever (along with just having <unknown> as filter) disappeared after:
akonadictl stop
akonadictl start

[Kde-bugs-3 (kde-bugs-3)]

Renè, the workaround from comment #118 still works for me: Add a filter like "From contains @" and let it execute "/usr/bin/sleep 0.5" (the time you need may vary, 0.3 wasn't enough for me).

[Hussam-v (hussam-v)]

(In reply to Christian Boltz from comment #139)
> Renè, the workaround from comment #118 still works for me: Add a filter like
> "From contains @" and let it execute "/usr/bin/sleep 0.5" (the time you need
> may vary, 0.3 wasn't enough for me).

Should I add that at the top so it gets executed before other filters?
Thank you.

[Tobias-leupold (tobias-leupold)]

I have it as my first filter, executed on each incoming mail (but I'm also using POP3, not IMAP – just in case this has an effect).

[Hussam-v (hussam-v)]

That breaks execution of other filters. I want it to still get filtered to
Inbox/Bugs/KDE

[René (rkrell)]

(In reply to Hussam Al-Tayeb from comment #142)
> That breaks execution of other filters. I want it to still get filtered to
> Inbox/Bugs/KDE

It should not break any other filters if you uncheck Advanced -> "If this filter matches stop processing here".
I created such a filter with the simple condition "All messages" and give it a try.
Thanks for the hint. Should be seriously solved anyway.

[Eikesauer (eikesauer)]

Created attachment 94721
signature.asc

Am Donnerstag, 24. September 2015, 17:06:58 schrieben Sie:
> https://bugs.kde.org/show_bug.cgi?id=283682
>
> --- Comment #142 from Hussam Al-Tayeb <email address hidden> ---
> That breaks execution of other filters. I want it to still get filtered to
> Inbox/Bugs/KDE

Obviously you need to deactivate "stop here if filter applies" on the extended
tab.

[J-diel (j-diel)]

Created attachment 94779
attachment-14215-0.dat

Dear,

I've a case of a mail, which reproducibly and consistently creates duplicates,
and a lot of them.
When I try to remove duplicates (<CTRL> *>), I get an error message.
I've attached the screenshot of the error message and the mail file itself
(after anonymising my mail address), maybe that helps for further analysis.
This mail file resides here:
~/.local/share/akonadi_maildir_resource_0/inbox/new/
and the creation of duplicates ends only, when I manually delete this file.

Am Freitag, 25. September 2015, 17:23:49 schrieben Sie:
> https://bugs.kde.org/show_bug.cgi?id=283682
>
> --- Comment #144 from Eike Sauer <email address hidden> ---
>
> Am Donnerstag, 24. September 2015, 17:06:58 schrieben Sie:
> > https://bugs.kde.org/show_bug.cgi?id=283682
> >
> > --- Comment #142 from Hussam Al-Tayeb <email address hidden> ---
> > That breaks execution of other filters. I want it to still get filtered to
> > Inbox/Bugs/KDE
>
> Obviously you need to deactivate "stop here if filter applies" on the
> extended tab.

[J-diel (j-diel)]

Created attachment 94780
duplicateMailExample.png

[Graeme Hewson (ghewson)]

It's best to put error messages in text form for easy reference.

The title of the popup message is "Fehler beim Löschen von doppelten Nachrichten – Kontact", which Google translates as "Failed to delete duplicate messages".

The content of the message is "Unable to fetch item from backend (collection 186) : Unable to retrieve item from resource: Ungļltigen Eintrag erhalten".

[J-andy-w (j-andy-w)]

Hello, I'm getting this problem on KMail Version 4.14.9
The 'sleep 0.5' command workaround didn't work for me.
However I have found that it only seems to happen when moving messages into mail folders that I imported (via Evolution 3 import from my previous maildir files.)

If I create a new folder and filter into that then the new messages move OK and stop reappearing back in the Inbox.

The problem also occurs on a manual move into an imported mail folder - the mail moves OK but pops up again in the Inbox

But again this doesn't happen with a newly created mail folder

Let me know if you want any debug information from these 'funny' folders - let me know what I need to do.
Thanks

[3-eric-x (3-eric-x)]

Filtering issues disappeared with kontact version 5.0.2 shipped with Kubuntu 15.10

[Wolfgang-mader (wolfgang-mader)]

I experienced the described issue for all akonadi/kmail versions which are ported to frameworks. The system I am using is Arch with the following (relevant) packages installed

- plasma-desktop: 5.5.1
- plasma-framework: 5.17
- framework components 5.17
- kde applications: 15.12, including kmail and akonadi

Moreover, as a dependency of digikam, the following kde 4 stuff is there
- kdebase-runtime: 15.12
- kdebase-lib: 15.12
- kdepimlibs4: 4.14.10
- libkipi4: 15.08
- libakonadi-qt4: 1.13.0
- libbaloo4: 4.14.3

I configure two generic imap resource in kmail. The first points to the mail server of my university, the second to the one of my web provider. I do not know what imap server they are running, but likely they are running different software. Thus, I guess, the issue is not with the server but with the client side. I do not configure anything special like server side subscription or sieve script.

On initial population of the mailboxes, everything seem fine. All messages are there and readable. The trouble starts as soon as I start to move around mails manually or via kmail filters or delete messages. Say, I move a message from the Inbox to the Private folder. The message disappears in the Inbox and seems to be moved to Privat. However, the message is not really moved. I is still in the Inbox and reappears the next time kmail refreshes the Inbox. The mail is grayed in the Inbox (everything is good with the messages in Private). In the Inboxr, I can not click on it, move or delete it. It just sits there, grayed out, not touchable. Now, using the web interface of my universities mail system, the mail is marked as deleted in the Inbox, but still there. At least, using the web interface, I can select the massage and delete it for real. Then, also in kmail, the message disappears. Unfortunately, I can not properly name the state of the deleted message in "imap"-terms.

On top of this issue, moved messages start to appear in the target folder multiple times. There is always one instance of the message which is clickable and good, but with time, multiple grayed out versions start to accumulate. They seem to be in the same state as the moved one in the source folder.

If there is any more information I can provide, do not hesitate to drop me a mail.

Reproducible: Always Steps to Reproduce:
1. Create a Imap resource in framework based kmail
2. Move or delete messages
3. Refresh mailbox and observe grayed out, uncklickable messages

Actual Results: Messages are not moved or deleted from mailbox.

Expected Results: Messages should be moved or deleted from mailbox.

[Wolfgang-mader (wolfgang-mader)]

Dear devs,

would if help if you had access to a mailbox showing this issue. If so, let me know, and I see what I can do.

[Wolfgang-mader (wolfgang-mader)]

I see the same bug in trojita. Do trojita and kmail2 share code?

[Răpițeanu Viorel-Cătălin (rapiteanu-catalin)]

I'm having the same problem on Kontact 5.1.1. For every filtered mail I receive, I get 19 more duplicates. This is getting absurd.

[Z-mike-y (z-mike-y)]

I just got hit by this.... Duplicated 17,000 messages, all did was hit the "apply all filters on folder" button, and then noticed that my phone was freaking out due to all the dupes.

kmail-libs-4.14.10-10.fc23.x86_64
baloo-akonadi-4.14.3-5.fc23.x86_64
akonadi-mysql-1.13.0-20.fc23.x86_64
akonadi-1.13.0-20.fc23.x86_64
kdepimlibs-akonadi-4.14.10-8.fc23.x86_64
kmail-4.14.10-10.fc23.x86_64

On Fedora 23.

Not happy, rest of the day is going to be fixing this mess

[Christian Hilberg (hilberg)]

Any updates on this regarding KF5-based KDEPIM 16.04? Has the issue been resolved?

[Dandress2 (dandress2)]

I had this bug a couple of years ago. It is more like a feature that can be turned on and off.

Tools>Configure KMail>Accounts>Select Account>Modify
On the Advanced tab check Automatically compact folders (expunges deleted messages)

[Christian Hilberg (hilberg)]

@Dennis: Hum, that setting does not change anything for me regarding dupes. Even when activated, the dupes appear.

Confirming that the bug persists for me (newly created IMAP kmail account, newly created filters) as of kmail 5.2.2, KDE SC 16.04.02, KF 5.23.0, on Debian/Sid.

[3-eric-x (3-eric-x)]

The activities, once you get used to use them do exactly what is missing. Unvotting for this bug and removing my email from the list.

[Christian Hilberg (hilberg)]

@Eric: While it might be true that activities do the trick for mail filtering, I guess we're talking about a bug in the existing Kmail2 mail filter functionality.

Testing with Kmail 5.2.3 (KDEPIM 16.04.3) on Debian/Sid shows a slightly different behaviour. Simple "move to" filters do not duplicate any filtered emails for me any longer.

However, messages filtered with the "pipe through" filter, as used by the anti-spam filters do get duplicated inside the destination folder if that folder resides on my IMAP server. If the destination folder is a local maildir, then the duplication does not occur.

[Christian Hilberg (hilberg)]

Okay, there is still some duplication in KMail 5.2.3 with the pipe-through ... if the filtered message is *not* spam, then it gets dupe'd inside the folder it resides, typically INBOX.

[Jwagner-5 (jwagner-5)]

Note that testing for a few hours might not be enough. Last time I tried, the bug only showed after 2-3 weeks of daily use. That's also why I'm hesitant about trying it again.

It would be nice if somebody with good knowledge of the inner working of KMail reviewed the workarounds suggested in this thread and posted a commented summary and recommendations for users (and possibly recommendations for developers).

[3-eric-x (3-eric-x)]

(In reply to Christian Hilberg from comment #160)
Hi Christian,
I am a bit embarrassed, because I answered the wrong bug report. Thus, my message has really nothing to do with this one!
Sorry about that.
Éric.

> @Eric: While it might be true that activities do the trick for mail
> filtering, I guess we're talking about a bug in the existing Kmail2 mail
> filter functionality.
>
> Testing with Kmail 5.2.3 (KDEPIM 16.04.3) on Debian/Sid shows a slightly
> different behaviour. Simple "move to" filters do not duplicate any filtered
> emails for me any longer.
>
> However, messages filtered with the "pipe through" filter, as used by the
> anti-spam filters do get duplicated inside the destination folder if that
> folder resides on my IMAP server. If the destination folder is a local
> maildir, then the duplication does not occur.

[Christian Hilberg (hilberg)]

(In reply to Eric Depagne from comment #163)
> (In reply to Christian Hilberg from comment #160)
> Hi Christian,
> I am a bit embarrassed, because I answered the wrong bug report. Thus, my
> message has really nothing to do with this one!

Seems there has been some confusion about local mail filtering inside Kmail and the general idea of "activities" (which I haven't fully grokked myself, still need to find my own use case for that). So...

> Sorry about that.

...nothing to worry about! :-)

> Éric.

Christian

> > @Eric: While it might be true that activities do the trick for mail
> > filtering, I guess we're talking about a bug in the existing Kmail2 mail
> > filter functionality.
> >
> > Testing with Kmail 5.2.3 (KDEPIM 16.04.3) on Debian/Sid shows a slightly
> > different behaviour. Simple "move to" filters do not duplicate any filtered
> > emails for me any longer.
> >
> > However, messages filtered with the "pipe through" filter, as used by the
> > anti-spam filters do get duplicated inside the destination folder if that
> > folder resides on my IMAP server. If the destination folder is a local
> > maildir, then the duplication does not occur.

[Christian Hilberg (hilberg)]

Update to behaviour of Kmail 5.2.3 (KDEPIM 16.04.3) on Debian/Testing:

Currently, I do not get any more dupes while pipe-through-filtering.

Changes in setup (now: working, before: dupes):

* IMAP now Dovecot (instead of ye olde Binc IMAP used before)
* Filtering now spamc+daemon (instead of spamassassin-perl before)

The spambayes filter is used in addition, which is the same as before.
Seems the bug could be triggered depending on the IMAP server implementation used. I tend to rule out the spamassassin-perl vs. smapc+daemon thing, as I've played with manually injecting sleeps at various steps in the filter processing, to no avail.

[J-andy-w (j-andy-w)]

Hello, it happens for me on mail coming in via POP so not IMAP related in my case - more details of my experience above on Comment 148 megalithic 2015-11-05 22:00:56 UTC

[Ianseeks (ianseeks)]

(In reply to megalithic from comment #166)
> Hello, it happens for me on mail coming in via POP so not IMAP related in my
> case - more details of my experience above on Comment 148 megalithic
> 2015-11-05 22:00:56 UTC

Try uncheck the "Apply filter to incoming messages" on each filter. Now just click on the inbox and click 'Folders/Apply all filters on Folder'. It has so far not produced a duplicate email for so its looking like filters on incoming emails is the culprit.

[Dvratil-7 (dvratil-7)]

*** Bug 322126 has been marked as a duplicate of this bug. ***

[Dvratil-7 (dvratil-7)]

*** Bug 377344 has been marked as a duplicate of this bug. ***

[Dvratil-7 (dvratil-7)]

*** Bug 322676 has been marked as a duplicate of this bug. ***

[Dvratil-7 (dvratil-7)]

*** Bug 293918 has been marked as a duplicate of this bug. ***

[Dvratil-7 (dvratil-7)]

*** Bug 323980 has been marked as a duplicate of this bug. ***

[Dvratil-7 (dvratil-7)]

*** Bug 362398 has been marked as a duplicate of this bug. ***

[Dvratil-7 (dvratil-7)]

*** Bug 333035 has been marked as a duplicate of this bug. ***

[Dvratil-7 (dvratil-7)]

*** Bug 346695 has been marked as a duplicate of this bug. ***

[J-andy-w (j-andy-w)]

(In reply to ianseeks from comment #167)
> (In reply to megalithic from comment #166)
> > Hello, it happens for me on mail coming in via POP so not IMAP related in my
> > case - more details of my experience above on Comment 148 megalithic
> > 2015-11-05 22:00:56 UTC
>
> Try uncheck the "Apply filter to incoming messages" on each filter. Now
> just click on the inbox and click 'Folders/Apply all filters on Folder'. It
> has so far not produced a duplicate email for so its looking like filters on
> incoming emails is the culprit.

Hello - if I do that my filters won't work (will they)? Do you want me to try a test?

In my case I only get duplicates when moving messages into some subfolders and not others. I also only get duplicates on some messages. I can send a screen shot of this. Interestingly the two duplicate/filtered messages come through with different file sizes. eg the first is 5.1KB and the second is 8.7KB. But they are the same message duplicated with the same time stamp. (not sent twice accidentally)

[Ianseeks (ianseeks)]

(In reply to megalithic from comment #176)
> (In reply to ianseeks from comment #167)
> > (In reply to megalithic from comment #166)
> > > Hello, it happens for me on mail coming in via POP so not IMAP related in my
> > > case - more details of my experience above on Comment 148 megalithic
> > > 2015-11-05 22:00:56 UTC
> >
> > Try uncheck the "Apply filter to incoming messages" on each filter. Now
> > just click on the inbox and click 'Folders/Apply all filters on Folder'. It
> > has so far not produced a duplicate email for so its looking like filters on
> > incoming emails is the culprit.
>
> Hello - if I do that my filters won't work (will they)? Do you want me to
> try a test?

The filters won't work as the emails come into the Inbox. Wait until the download of emails has finished then you click ont he Inbos foler to highlight it and then select "Apply All Filters on Folder" from the Folder menu (or add this option to the tool bar)

> In my case I only get duplicates when moving messages into some subfolders
> and not others. I also only get duplicates on some messages. I can send a
> screen shot of this. Interestingly the two duplicate/filtered messages come
> through with different file sizes. eg the first is 5.1KB and the second is
> 8.7KB. But they are the same message duplicated with the same time stamp.
> (not sent twice accidentally)

I didn't get duplicates on all emails either but i don't get any duplicates at all now because of the way i do it as described.
Its just temporary until the devs fix filtering, there are a few more problems with filtering other than duplication of emails.

[Ianseeks (ianseeks)]

Correction of last message typing screwup

"ont he Inbos foler" = on the Inbox folder

[Andreas Mahel (andreas-mahel)]

I'm experiencing email duplication after filter activity as well (openSUSE Tumbleweed, most recent versions).

The filters move mainly mailing list traffic from a POP3 account into local sub folders.

Duplication happens both for automatic filtering as well as executing the filters manually.

In most cases, the duplicated entry shows a different size than the original one (but not always).

On top, selecting any of the two entries will result in Kmail endlessly waiting for Akonadi.

A akonadictl fsck and vacuum does not remove the duplicates; also a "Remove duplicate emails" from the "Folder" menu does'nt, and ends with an error message.

In the meantime I have deleted the Akonadi database several times, as well as re-created the filters manually, to no avail.

As a workaround for myself I've written a short SQL script to show / delete the offending rows from the database (from within akonadiconsole) - this is tedious, but at least allows me to read all mails.

(How) can I gather more information about what exactly is going on? Both Kmail and Akonadi are kind of cryptic in their output - is there a way to increase verbosity?

[Ianseeks (ianseeks)]

(In reply to Andreas Mahel from comment #179)
> I'm experiencing email duplication after filter activity as well (openSUSE
> Tumbleweed, most recent versions).
>
> The filters move mainly mailing list traffic from a POP3 account into local
> sub folders.
>
> Duplication happens both for automatic filtering as well as executing the
> filters manually.
>
> In most cases, the duplicated entry shows a different size than the original
> one (but not always).
>
> On top, selecting any of the two entries will result in Kmail endlessly
> waiting for Akonadi.
>
> A akonadictl fsck and vacuum does not remove the duplicates; also a "Remove
> duplicate emails" from the "Folder" menu does'nt, and ends with an error
> message.
>
> In the meantime I have deleted the Akonadi database several times, as well
> as re-created the filters manually, to no avail.
>
> As a workaround for myself I've written a short SQL script to show / delete
> the offending rows from the database (from within akonadiconsole) - this is
> tedious, but at least allows me to read all mails.
>
> (How) can I gather more information about what exactly is going on? Both
> Kmail and Akonadi are kind of cryptic in their output - is there a way to
> increase verbosity?

Since I posted my change to manual filtering, i've only had 3 emails duplicated so there is still a problem but not as bad as it was on filtering on incoming messages.

[S-pamme-f (s-pamme-f)]

(In reply to Andreas Mahel from comment #179)
> I'm experiencing email duplication after filter activity as well (openSUSE
> Tumbleweed, most recent versions).
>
> The filters move mainly mailing list traffic from a POP3 account into local
> sub folders.
>
> Duplication happens both for automatic filtering as well as executing the
> filters manually.
>
> In most cases, the duplicated entry shows a different size than the original
> one (but not always).
>
> On top, selecting any of the two entries will result in Kmail endlessly
> waiting for Akonadi.
>
> A akonadictl fsck and vacuum does not remove the duplicates; also a "Remove
> duplicate emails" from the "Folder" menu does'nt, and ends with an error
> message.
>
> In the meantime I have deleted the Akonadi database several times, as well
> as re-created the filters manually, to no avail.
>
> As a workaround for myself I've written a short SQL script to show / delete
> the offending rows from the database (from within akonadiconsole) - this is
> tedious, but at least allows me to read all mails.
>
> (How) can I gather more information about what exactly is going on? Both
> Kmail and Akonadi are kind of cryptic in their output - is there a way to
> increase verbosity?

Exactly as you describe. Sometimes I can get things going again for a short time by doing a restart of akonadi, but lately more often than not this causes akonadi to crash.

Weird thing though; in some sub-folders the "delete duplicates" functionality works fine, in other sub-folders (underneath the same account) it will indeed hang everything solid with an error message at the end, just like you describe.

[Kollix (kollix)]

Just for the records, I once tried to dig into the Akonadi code to find the bug.
I put my analysis to the other bug report, which is now set to a duplicate of this.
https://bugs.kde.org/show_bug.cgi?id=362398#c5
Just that it does not get lost, and I still think my findings are correct, here is it again:

I'm hunting the "duplicate mail bug" and I think I've found a bug in the Akonadi Server.
What happens here is the following:
- a resource adds a new mail item (id=3, collection=17)
- Akonadi sends "Item added" to the filter agent
- filter agent sends a "move" command to move item id=3 from collection=17 to collection=19
- Akonadi sends "item added" to the maildir resource (this is the first "added" message from the creation, so the
  item still has id=3 collection=17)
- I'd say in the Akonadi DB the collectionId of item=3 is now 19 (after the move)
- Now the maildir resource gets triggered by its internal file watcher and triggers a sync
- maildir gets called with retrieveItems and lists what it has on disc, which is the item (id=3) in the collection=17
- Akonadi receives this and inside Merge::parseStream() there the bad thing happens:
  It hits this "if" branch (server/src/handler/merge.cpp Line 301)
      // No item with such GID/RID exists, so call AkAppend::insert() and behave
      // like if this was a new item
 and INSERTS the item AGAIN into collection=17 with a new id=4
 Now I have the same mail item twice in the DB
 Since the maildir "itemsRetrieved" answer contains for the first time the RID, the newly inserted item gets this RID,
 and the previously intserted one id=3 does not and is left alone with a NULL RID

Someone more knowledgeable about the Akonadi internals should have a look, please.

[Bingmybong (bingmybong)]

On Wednesday, 5 April 2017 08:53:00 BST you wrote:
> https://bugs.kde.org/show_bug.cgi?id=283682
>
> --- Comment #182 from Martin Koller <email address hidden> ---
> Just for the records, I once tried to dig into the Akonadi code to find the
> bug.
> I put my analysis to the other bug report, which is now set to a duplicate
> of this.
> https://bugs.kde.org/show_bug.cgi?id=362398#c5
> Just that it does not get lost, and I still think my findings are correct,
> here is it again:
>
> I'm hunting the "duplicate mail bug" and I think I've found a bug in the
> Akonadi Server.
> What happens here is the following:
> - a resource adds a new mail item (id=3, collection=17)
> - Akonadi sends "Item added" to the filter agent
> - filter agent sends a "move" command to move item id=3 from collection=17
> to collection=19
> - Akonadi sends "item added" to the maildir resource (this is the first
> "added" message from the creation, so the
> item still has id=3 collection=17)
> - I'd say in the Akonadi DB the collectionId of item=3 is now 19 (after the
> move)
> - Now the maildir resource gets triggered by its internal file watcher and
> triggers a sync
> - maildir gets called with retrieveItems and lists what it has on disc,
> which is the item (id=3) in the collection=17
> - Akonadi receives this and inside Merge::parseStream() there the bad thing
> happens:
> It hits this "if" branch (server/src/handler/merge.cpp Line 301)
> // No item with such GID/RID exists, so call AkAppend::insert() and
> behave
> // like if this was a new item
> and INSERTS the item AGAIN into collection=17 with a new id=4
> Now I have the same mail item twice in the DB
> Since the maildir "itemsRetrieved" answer contains for the first time the
> RID, the newly inserted item gets this RID,
> and the previously intserted one id=3 does not and is left alone with a
> NULL RID
>
>
> Someone more knowledgeable about the Akonadi internals should have a look,
> please.

Good work.
You could post this to kdepim-users list as Daniel Vrátil, who i think is the
prime dev on akonadi, appears to frequent this list. Or if you are not a
member i can forward for you.

[Kollix (kollix)]

In fact I sent this information back then directly to Daniel but he obviously did not have time to look into it or does not see it as important.

[Bll-guayam-gj5 (bll-guayam-gj5)]

(In reply to Martin Koller from comment #182)

Martin... if that's the case, then this "duplicates" issue would _only_ happen for messages being auto-filtered, correct? Instead, if messages were first picked up and dropped into a folder and then _manually_ filtered, the events would reorder such that the "trigger the filter, and move the msg" doesn't happen until after the maildir resource had finished accepting the message.

(just wrapping my brain around it, and wanted to confirm that the above sounds reasonable)

[Jwagner-5 (jwagner-5)]

As to "manually filtered" (comment #185), please distinguish between "manually applied filters (menu: Message --> Apply Filter)" and "manually filtered by moving messages between folders (menu: Message --> Move Message To)".

Some comments report duplication even in the later case.

[Andreas Mahel (andreas-mahel)]

As per my comment above (comment #179), it would be "running the filters manually (via Ctrl-J)". I haven't yet reverted to not using filters at all and dispatching several mailing lists to their respective sub folders by using the "Move to folder" functionality.

I can also confirm that automatic filtering leads to considerably more duplicate entries than manual filtering.

I have set up a VM with a simple mail setup (one POP3 resource receiving messages from 3 mailing lists), and with automatic filtering, the issue was clearly visible with about 10 to 20% of mails affected. I have reset that one now, and switched to manual filtering, to see if I catch any duplicates here as well.

[Ianseeks (ianseeks)]

For me, manual filtering means using the "Apply All Filters on Folder" menu item or as i have done, put it as a button on the toolbar.

[J-andy-w (j-andy-w)]

I've switched to filtering after the messages come in by clicking 'Folder - Apply All filters on Folder' on the Inbox.

Just to confirm this has totally stopped the duplication in my case.
Great to see we might be getting somewhere on this - thanks folks

[S-pamme-f (s-pamme-f)]

But is one of the developers actually looking at this? The duplication of the messages, combined with the akonadi hanging itself out to dry when deleting messages (resulting in the famous "retrieving folder content" screen) makes KMail basically unusable. I'm really baffled how such a crucial piece of software (unless developers use other program to read email) can be so broken for so long.

[Kollix (kollix)]

I am currently looking into this and have already implemented some patches which hopefully improve the situation (although I have no final fix yet).
E.g. https://phabricator.kde.org/D5667
It's very complicated and has to do with timing of messages between 3 processes ...

[S-pamme-f (s-pamme-f)]

Excellent! Good to know there is some progress.

[Bll-guayam-gj5 (bll-guayam-gj5)]

Agreed! Exciting to see progress, as KMail is close to unusable at this point.

MUCH appreciated!

[Bingmybong (bingmybong)]

(In reply to Martin Koller from comment #191)
> I am currently looking into this and have already implemented some patches
> which hopefully improve the situation (although I have no final fix yet).
> E.g. https://phabricator.kde.org/D5667
> It's very complicated and has to do with timing of messages between 3
> processes ...

Thanks Martin, your work on this and other bugs will be most appreciated by lots of users

[Alexandre-bonneau (alexandre-bonneau)]

Kmail version 5.2.3 (in Debian Stable) is definitely showing this bug (see the attached screenshot).

[Alexandre-bonneau (alexandre-bonneau)]

Created attachment 107642
Duplicated mails in Kmail 5.2.3

[Kde-v (kde-v)]

Just hit the same. I noted that the duplicated messages, were messages never fully moved in the first place. They ghosted in the original folder, and the ghosts were copied.

The folders this happens to also does not have working folder properties. Nothing you change in folder properties have ANY effect and are immediately lost

[Árpád Lukács (lukacs-arpad)]

With upgrading to version 5.5.3 on Kmail, the duplication problem got worse in my case.
I have a rather large (~ 10000) collection of messages. It happens after filtering, it can be "fixed" by clearing akonadi cache with akonadiconsole, but it appears again.
Adding a filter on top of the filter list, that executes "/usr/bin/sleep 15" did not help. I could not find out if it happens with the folders that contain many messages, or in all of them (i think the latter).
What did change with the new version (compared to the one there was in kubuntu 14.04) is that now I get an error message when trying to access the folder with the duplicate messages, that kmail cannot fetch the message because of multiple candidates. Previously, i could erase one.

[DeMus (jan-mussche)]

I can confirm this bug with both an IMAP and a POP account. Mails are sometimes (not always, filters don't always work well) transferred from the inbox to the correct folder and a little bit later the mails are also back in the inbox. This can go on forever.
When I delete them from the inbox it is over and they don't appear anymore.

[RogerL (roger-larsson)]

I have also seen this (and my wife on her setup - similar to mine)
and I wonder if some of this is due to the huge number of options and the more complex environment.

I want to be able to read mails both on my computer and on my mobile.
I want the computer to be the primary storage of my mails.
POP/IMAP account - using IMAP for both

Computer
- copied new messages locally using filters into correct folder
- filters that mark a message, resulted in two copies (one shadow - not readable) the updated state could be seen from mobile too...
- had "download for offline processing" enabled (trying with it off now)
   noticed that it
- push on
- recheck 5 minutes
...

I wonder if my settings caused numerous rereads, store on disk with different generations of each mail...

-rw-r--r-- 1 roger users 4589 11 mar 00.20 /home/roger/.local/share/akonadi/file_db_data/99/822899_r0
-rw-r--r-- 1 roger users 4100 6 apr 18.29 /home/roger/.local/share/akonadi/file_db_data/99/827899_r61
-rw-r--r-- 1 roger users 6372 6 apr 18.29 /home/roger/.local/share/akonadi/file_db_data/99/828199_r1360
-rw-r--r-- 1 roger users 4348 6 apr 18.30 /home/roger/.local/share/akonadi/file_db_data/99/831799_r1359
-rw-r--r-- 1 roger users 4126 6 apr 18.30 /home/roger/.local/share/akonadi/file_db_data/99/833599_r61

[Psychonaut (psychonaut)]

Is this problem still reproducible in the latest version of KMail? There haven't been any comments on this bug in over a year; I'm wondering if that's because the problem has been fixed or because it has driven most of the user base away. I'd love to return to using KMail but can't as long as this message duplication problem persists. (I was getting dozens of duplicated messages every day, which was too much to put up with.)

[3-eric-x (3-eric-x)]

(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail? There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away. I'd love to return to using KMail but can't as long as
> this message duplication problem persists. (I was getting dozens of
> duplicated messages every day, which was too much to put up with.)

It has been fixed I believe, since it disappeared from my kmail in 2015.

[Psychonaut (psychonaut)]

(In reply to Eric Depagne from comment #203)

> It has been fixed I believe, since it disappeared from my kmail in 2015.

And yet there are plenty of comments here from 2015 to 2018 claiming the problem still exists. I would prefer to hear from someone who was experiencing the problem more recently (say, 2018) who can confirm that it is no longer reproducible.

[Pe2er (pe2er)]

(In reply to Eric Depagne from comment #203)
> (In reply to Tristan Miller from comment #202)
> > Is this problem still reproducible in the latest version of KMail? There
> > haven't been any comments on this bug in over a year; I'm wondering if
> > that's because the problem has been fixed or because it has driven most of
> > the user base away. I'd love to return to using KMail but can't as long as
> > this message duplication problem persists. (I was getting dozens of
> > duplicated messages every day, which was too much to put up with.)
>
> It has been fixed I believe, since it disappeared from my kmail in 2015.

It certainly hasn't. I still get numerous duplicates daily on this Gentoo box.

I think the comments dried up because (a) there was nothing new to add; (b) users grew tired of banging on to no avail.

[Árpád Lukács (lukacs-arpad)]

I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3), that came with Ubuntu 18.04. Then I started using the workaround from comment #118.

[Psychonaut (psychonaut)]

(In reply to Árpi from comment #206)
> I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3),
> that came with Ubuntu 18.04. Then I started using the workaround from
> comment #118.

Lucky you; that workaround did nothing for me.

Is there some sort of bug bounty program for KDE? If so I wouldn't mind chipping in €20 or so towards a fix for this issue.

[Graeme Hewson (ghewson)]

I haven't seen the problem for a while, maybe two years. I use POP3 only, not IMAP. I run Kubuntu; currently I'm on 19.04.

Sometimes kmail crashes, and I wonder if the way I clean up is related to my not seeing duplicate filtered messages. When kmail crashes, I stop akonadi ("akonadictl stop") and look in the maildir structure like this:

cd .local/share/.local-mail.directory
ls */new/*S

The S indicates messages which have been Seen, and according to the Maildir specification the files should not be in a"new" subdirectory; they should have been moved to a "cur" subdirectory. I therefore move them while akonadi is stopped. E.g.,

mv -i wastebin/new/*S wastebin/cur

I then run "akonadictl fsck" and restart akonadi ("akonadictl start").

As I say, it's pure speculation that this is why I don't see duplicate messages now.

[Graeme Hewson (ghewson)]

Apologies.

akonadictl start
akonadictl fsck

is the correct order.

[Alexandre-bonneau (alexandre-bonneau)]

On my end, I'm currently using Kmail 5.9.3 (Debian buster), and am seeing no duplicates using imap only.
Some very rare times I can see a few duplicates, but they disappear by themselves pretty quicly.

[Jwagner-5 (jwagner-5)]

(In reply to Tristan Miller from comment #202)
> [...] because the problem has been fixed or because it has driven most of
> the user base away. I'd love to return to using KMail [...]

I didn't stop using KMail. I stopped using filters.

If I find time to explore alternatives and find something suitable I may switch. Also possible I will use two mail clients in parallel.

Coincidentally, I am well positioned to test alternatives as I moved away from using folders under "Local Folders" to using a local imap server a while ago (because, at that time, KMail stored messages in a database instead of a maildir). This should make it easy to use multiple clients in parallel.

[S-pamme-f (s-pamme-f)]

(In reply to Eric Depagne from comment #203)
> (In reply to Tristan Miller from comment #202)
> > Is this problem still reproducible in the latest version of KMail? There
> > haven't been any comments on this bug in over a year; I'm wondering if
> > that's because the problem has been fixed or because it has driven most of
> > the user base away. I'd love to return to using KMail but can't as long as
> > this message duplication problem persists. (I was getting dozens of
> > duplicated messages every day, which was too much to put up with.)
>
> It has been fixed I believe, since it disappeared from my kmail in 2015.

Most definitely not fixed. Still run into it multiple times per day with some messages showing up 5 or more times. Sometimes I just delete the entire folder and filter and just set everything up again. Sometimes seems to do the trick. I have been unable to find any logic behind why some messages get duplicated where others don't (in the same folder).

[Árpád Lukács (lukacs-arpad)]

(In reply to Tristan Miller from comment #207)
> (In reply to Árpi from comment #206)
> > I did get numerous duplicates when I installed kmail 5.7.x (I think, x=3),
> > that came with Ubuntu 18.04. Then I started using the workaround from
> > comment #118.
>
> Lucky you; that workaround did nothing for me.
>
> Is there some sort of bug bounty program for KDE? If so I wouldn't mind
> chipping in €20 or so towards a fix for this issue.

I checked my filters; what I am doing is not exactly as described in comment #118, but instead, I added an extra action, to execute
/bin/sleep 1
to each of my filters. Since then, no duplicates.

[Tobias-leupold-u (tobias-leupold-u)]

I've been using IMAP for some years now, and I don't get duplicates anymore when filtering. In the case my filters ever do filter, which is roughly the case in 80 % of the mails they should work on (otherwise, nothing happens).

After almost eight years and more than 200 comments, I think one should rather wonder if this even can be fixed inside this Akonadi monster I personally wouldn't even dare to look at the code, or if one should reconsider the whole thing.

Only thing I can't understand is that the KMail guys are not annoyed themselves by this ... I mean, I don't think they use Trojitá, do they?! But no harm meant,
remarkably enough I still stick to KMail ;-)

[Bingmybong (bingmybong)]

(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail? There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away. I'd love to return to using KMail but can't as long as
> this message duplication problem persists. (I was getting dozens of
> duplicated messages every day, which was too much to put up with.)

The way i reduced the duplicates to a point where it hardly happens (maybe once a week) is to stop filtering on incoming messages and only apply the filters manually once all emails are in. I'm using IMAP but I think the POP3 still has a lot of issues.

[Bll-guayam-gj5 (bll-guayam-gj5)]

(In reply to Tristan Miller from comment #202)
> Is this problem still reproducible in the latest version of KMail? There
> haven't been any comments on this bug in over a year; I'm wondering if
> that's because the problem has been fixed or because it has driven most of
> the user base away.

Driven away.

Was a KMail for what seems like forever (~2000?) (I've been a KDE user since it first came out in '96), but after dealing with workarounds and hacks for two+ years I eventually gave in and abandoned KMail back in mid-2017.

Took me multiple weeks to get all of my mail (~12GB) imported into Thunderbird, get my 200+ mail filters set up again, and to remove all of the duplicates and junk that various KMail bugs had done to my mail. Was the most painful thing I'd done in years, and I knew the switch was going to suck bad so I held out as long as I could.

KMail eventually just got unusable, period. I'd have to do workarounds and restart akonadi every single time I picked up my POP mail, which just made KMail effectively useless. After realizing I was using Alpine to read all of my new mail and delete junk _before_ fetching it into KMail, I finally gave in and switched.

LOVED KMail, but THIS bug led me to finally throw in the towel and give it up.

[Ederag (ederag)]

(In reply to graham from comment #216)
> Was a KMail for what seems like forever (~2000?) (I've been a KDE user since
> it first came out in '96), but after dealing with workarounds and hacks for
> two+ years I eventually gave in and abandoned KMail back in mid-2017.

There was a commit in 2017-05-04
https://phabricator.kde.org/R206:43f2cde61f98317eb13d98222a57bc6df323a308
that was about this bug.
And indeed in openSUSE Leap-15 (kdepim-runtime 17.12.3),
there are much fewer duplicates than before.

There is now a related bug about unreachable duplicates
that has already been reported as #388869.

(In reply to Tobias Leupold from comment #214)
> I think one should rather wonder if this even can be fixed
> inside this Akonadi monster I personally wouldn't even dare to look at
> the code, or if one should reconsider the whole thing.

kmail/akonadi handle my > 15GB maildir gracefully,
which is quite a performance. Congratulations !

So it is not perfect, but to me it is an amazing piece of software.
Please consider that it is very hard for maintainers
to fix a bug that is not reproducible on their setup.

[Ederag (ederag)]

With a working link: bug #388869

[Pe2er (pe2er)]

> Please consider that it is very hard for maintainers
> to fix a bug that is not reproducible on their setup.

True, but there's nothing to stop them from enlisting help from knowledgeable users (at least a few have offered, including me). They won't do it though; I can only speculate as to why not.

[Tom (thomasmca)]

This bug is happening to me using KMail 5.10.3 on Lubuntu 19.04. Four emails that get moved into a folder via a filter get downloaded into that folder over and over. I deleted 20+ copies of each message recently, and now there are 8 more of each one.

I am downloading via POP3 from GMail.

[Linuxfan-2 (linuxfan-2)]

Version KMail 5.13.3 (19.12.3)
OS Gentoo Linux

I have been plagued by this bug for a very long time. Today I found out, that for ages there has been an error message in my mysql/mariadb log-file:

[ERROR] Incorrect definition of table mysql.event: expected column 'sql_mode' at position 14 to have type set ...

This is the first time I saw this error, as there is no other application than akonadi using mariadb on my pc and akonadi was never complaining. This mariadb-error could be fixed by running mysql_upgrade (google it to find an explanation, this is due to an incomplete upgrade of the mariadb-package)

After fixing the database, kmail has not been showing duplicated mails any longer. Also filters have been working flawlessly.

[Ederag (ederag)]

(In reply to linuxfan from comment #220)
> for ages there has been an error message in my mysql/mariadb log-file:
>
> [ERROR] Incorrect definition of table mysql.event: expected column
> 'sql_mode' at position 14 to have type set ...

Thanks for sharing.
In which log file exactly did you find this error ?

/var/log/mysql/ is empty here, so

pgrep -af mysqld
/usr/sbin/mysqld --defaults-file=/home/ederag/.local/share/akonadi/mysql.conf --datadir=/home/ederag/.local/share/akonadi/db_data/ --socket=/tmp/akonadi-ederag.Yqlbg1/mysql.socket --pid-file=/tmp/akonadi-ederag.Yqlbg1/mysql.pid

grep -1 "error log" /home/ederag/.local/share/akonadi/mysql.conf

# # error log file name, relative to datadir (default:hostname.err)
log_error=mysql.err

grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
<nothing>

That was with the "1 s" delay filter active,
just deactivated; we'll see.

[Pe2er (pe2er)]

(In reply to ederag from comment #221)

> grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
> <nothing>

Here, I see:

$ grep -i error .local/share/akonadi/db_data/mysql.err
2020-05-08 13:12:10 0 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
2020-05-08 13:12:10 9 [ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not found.

I don't know how long that's been happening, because I haven't had any duplicates for quite a long time. I remember, though, that when I was getting them, that error log had numerous errors of various types.

[Bingmybong (bingmybong)]

(In reply to Peter Humphrey from comment #222)
> (In reply to ederag from comment #221)
>
> > grep -i error /home/ederag/.local/share/akonadi/db_data/mysql.err
> > <nothing>
>
> Here, I see:
>
> $ grep -i error .local/share/akonadi/db_data/mysql.err
> 2020-05-08 13:12:10 0 [ERROR] Can't open and lock privilege tables: Table
> 'mysql.servers' doesn't exist
> 2020-05-08 13:12:10 9 [ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not
> found.
>
> I don't know how long that's been happening, because I haven't had any
> duplicates for quite a long time. I remember, though, that when I was
> getting them, that error log had numerous errors of various types.

I've been using postgresql and i've hardly seen any since but as of today i got duplicates by the bucket load.

opensuse:tumbleweed:20200506
Qt: 5.14.1 KDE Frameworks: 5.69.0 - KDE Plasma: 5.18.4 - kwin 5.18.4
kmail2 5.14.0 (20.04.0) - akonadiserver 5.14.0 (20.04.0) -

[Linuxfan-2 (linuxfan-2)]

the log file is /var/log/mysql/mysqld.err

However, digging deeper, the problem might be related to two instances of mariadb being present.

pgrep -af mysqld yields

2402 /usr/sbin/mysqld --defaults-file=/etc/mysql/my.cnf
2881 /usr/sbin/mysqld --defaults-file=/home/linuxfan/.local/share/akonadi/mysql.conf --datadir=/home/linuxfan/.local/share/akonadi/db_data/ --socket=/run/user/10001/akonadi/mysql.socket --pid-file=/run/user/1000/akonadi/mysql.pid

While /var/log/mysql/mysqld.err is now clear, in

/home/linuxfan/.local/share/akonadi/db_data/mysql.err

I can see two further errors and some warnings:

[ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist
 [Warning] Can't open and lock time zone table: Table 'mysql.time_zone_leap_second' doesn't exist trying to live without them
 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")
 [Note] Reading of all Master_info entries succeeded
 [Note] Added new Master_info '' to hash table
 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.4.12-MariaDB' socket: '/run/user/1000/akonadi/mysql.socket' port: 0 Source distribution
[ERROR] InnoDB: Table `mysql`.`innodb_table_stats` not found.

what is more important, I believe, is that akonadi seems to start the second mysql instance but some of the data is actually going to my main mysql server instead and its data directory /home/mysql_data/ while other parts are ending up in the akonadi-instance of mysql.

After disabling the stand-alone server, I found that ~/.config/akonadi/akonadiserverrc

still contained the line

Options="UNIX_SOCKET=/var/run/mysql/mysqld.socket"

pointing to the main server socket. Clearing this up, everything seems to end up in the akonadi-version of mysql (with the above warnings/errors, not sure what that means). I am unsure, why some of the data still went to the akonadi-version of mysql, though.

From my point of view, I seem to have had some mess in the config files as I have been dragging/upgrading my /home/ directory with me since 1997 .... (and yes, I have been using KDE ever since the original beta1)

[Ederag (ederag)]

Well, only half a day before a duplicate is back.
(again, from lists.gnu.org)
Same status as in comment #221.
And nothing suspicious in mysql.err (no ERROR).

[Alexandre-bonneau (alexandre-bonneau)]

(In reply to Alexandre Bonneau from comment #210)
> On my end, I'm currently using Kmail 5.9.3 (Debian buster), and am seeing no
> duplicates using imap only.
> Some very rare times I can see a few duplicates, but they disappear by
> themselves pretty quicly.

For info (I posted that info in another related issue already), I found out that I only get duplicates when I have 2 computers running akonadi-server.

For instance it happens when I'm on my main rig, then casually open my laptop. Then mails are duplicated a lot...until I close one of the akonadi server (usually the one from the laptop).

[Ederag (ederag)]

(In reply to Alexandre Bonneau from comment #226)
> For info (I posted that info in another related issue already), I found out
> that I only get duplicates when I have 2 computers running akonadi-server.

Here also, two computers once concurrently ran akonadi
with the same configuration (synchronized with unison).
But duplicates do appear, even when the other akonadi is stopped.
(pgrep akonadi yields nothing on the other computer)
[akonadictl 5.16.1 (20.12.0)]

[R-kunschke (r-kunschke)]

The problem is still present in kmail for a long time. I have filters for using bogofilter in kmail.
Is there any workaround or something? Or a filter for kmail which removes all duplicates?
And is there any explanation why this is happenning?

[Ederag (ederag)]

(In reply to r.kunschke from comment #228)
> Is there any workaround or something?

Adding this filter helped a lot here:
- match all messages
- Executed Command: /usr/bin/sleep 1,

[Christian Hilberg (hilberg)]

Created attachment 148406
signature.asc

Am Mittwoch, 27. April 2022, 10:57:26 CEST schrieb ederag:
> https://bugs.kde.org/show_bug.cgi?id=283682
>
> --- Comment #229 from ederag <email address hidden> ---
> (In reply to r.kunschke from comment #228)
> > Is there any workaround or something?
>
> Adding this filter helped a lot here:
> - match all messages
> - Executed Command: /usr/bin/sleep 1,

Depending on how quick the IMAP server is, this had
little to no effect for me when I tried, so YMMV.

(bye)^2

[Bingmybong (bingmybong)]

(In reply to r.kunschke from comment #228)
> The problem is still present in kmail for a long time. I have filters for
> using bogofilter in kmail.
> Is there any workaround or something? Or a filter for kmail which removes
> all duplicates?
> And is there any explanation why this is happenning?

There is an option on the "Folder" menu to remove duplicates that works reasonably well.
I used to have this issue but its now pretty much gone since i changed from 1. POP3 to Imap and 2. Mysql to Postgresql for the backend, its very very rare for me to get a duplicate now

[Christoph Thielecke (crissi99)]

Still valid in akonadi 21.12.3 on Kubuntu 22.04.

I think akonadi should automatically handle this problem.

[Pe2er (pe2er)]

New symptom.

I have a local server that fetches POP3 mail from my ISP and serves it as IMAP4. I've had only rare duplicates since setting this arrangement up a few years ago.

I discovered yesterday that creating a KMail POP3 account led immediately to numerous duplicates, as half-expected, but the key point here is that duplicates kept occurring, even after I'd deleted the POP3 account and rebooted.

This is KMail version 5.20.3 (22.04.3) on Gentoo stable, Plasma version 5.24.6-r1. I tried postgres several years ago but it didn't improve anything, so I'm sticking with mariadb.

[Alexander Neundorf (neundorf)]

Here on a current openSuse Tumbleweed it happens too, akonadi 5.22.1 (22.12.1).
Or at least the old duplicates are still here, not sure whether new ones have appeared this week already.

[Dcriego177 (dcriego177)]

Comment on attachment 74538
filter log, a duplicated message is fetched

><html>
><body>
><p>------------------------------</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;Subject&quot; &lt;contains&gt; &quot;Dit kreditkort blev brugt pￜ bￜrnepornografi site&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;Subject&quot; &lt;contains&gt; &quot;Dit kreditkort blev brugt pï¿œ bï¿œrnepornografi site&quot; (<i>Donación de la Hermana Alice Jahab</i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;List-Id&quot; &lt;contains&gt; &quot;&lt;nepomuk.kde.org&gt;&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;List-Id&quot; &lt;contains&gt; &quot;&lt;nepomuk.kde.org&gt;&quot; (<i></i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;Subject&quot; &lt;contains&gt; &quot;Beskyt dit Visa kort ved internethandel&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;Subject&quot; &lt;contains&gt; &quot;Beskyt dit Visa kort ved internethandel&quot; (<i>Donación de la Hermana Alice Jahab</i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;From&quot; &lt;contains&gt; &quot;<email address hidden>&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;From&quot; &lt;contains&gt; &quot;<email address hidden>&quot; (<i>alice jahab &lt;<email address hidden>&gt;</i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;Subject&quot; &lt;contains&gt; &quot;aktivering af nye sikkerheds&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;Subject&quot; &lt;contains&gt; &quot;aktivering af nye sikkerheds&quot; (<i>Donación de la Hermana Alice Jahab</i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;Subject&quot; &lt;contains&gt; &quot;venligst din opdatering&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;Subject&quot; &lt;contains&gt; &quot;venligst din opdatering&quot; (<i>Donación de la Hermana Alice Jahab</i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;List-Id&quot; &lt;contains&gt; &quot;&lt;openseamap-maps.lists.sourceforge.net&gt;&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;List-Id&quot; &lt;contains&gt; &quot;&lt;openseamap-maps.lists.sourceforge.net&gt;&quot; (<i></i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;List-Id&quot; &lt;contains&gt; &quot;&lt;marble-devel.kde.org&gt;&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;List-Id&quot; &lt;contains&gt; &quot;&lt;marble-devel.kde.org&gt;&quot; (<i></i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;List-Id&quot; &lt;contains&gt; &quot;&lt;kde-core-devel.kde.org&gt;&quot;</p>
><p>[11:48:19] <font color=#FF0000>0 = </font>&quot;List-Id&quot; &lt;contains&gt; &quot;&lt;kde-core-devel.kde.org&gt;&quot; (<i></i>)</p>
><p>[11:48:19] <b>Evaluerer filterregler:</b> (match all of the following)
> &quot;List-Id&quot; &lt;contains&gt; &quot;&lt;digikam-users.kd...

[Pe2er (pe2er)]

I think the real problem in KMail is in the handling of locally stored emails.

Since I installed a local server to fetch POP3 mails from my ISP and serve them to KMail as IMAP4, the problem has largely gone away.

The problem was still present at in interim stage when I was downloading all mails from my IMAP4 server, but now that I leave them on the server I don't see the problem any more.