Thunderbird incorrectly shows some Gmail archived messages as unread via IMAP

[Michał Gołębiowski-Owczarek]

Binary package hint: thunderbird

Steps to reproduce:
1) Wait for a few messages to arrive in Inbox, thus being marked as unread.
2) Mark all of them as read, then archive them.
3) Click on All Mail folder.

Actual Results:
Only one of recently archived messages is marked as read (usually the newest one), the rest must be marked as read again.

Expected Results:
All of recently marked as read in Inbox and archived mails should be marked as read in All Mail folder, too.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: thunderbird 3.0.5+build2+nobinonly-0ubuntu0.10.04.1
ProcVersionSignature: Ubuntu 2.6.32-23.37-generic 2.6.32.15+drm33.5
Uname: Linux 2.6.32-23-generic x86_64
NonfreeKernelModules: wl
Architecture: amd64
Date: Fri Jul 9 02:22:30 2010
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
SourcePackage: thunderbird

[Davec-ashesandwater (davec-ashesandwater)]

Some updates:

1 - This also occurs in safe mode.
2 - It's not an issue of the count, but that the messages subjects already read in Inbox and not marked unread in Gmail->All Mail are showing up in the new mail notifier.

So, as follows:

I get 3 messages. I read all three.
I get 1 new message. Popup notifier says 'You have 1 new message' but the subject preview below that lists the subjects for the 1 new message in the inbox and the subjects for all the messages previously read but still marked as unread in Gmail->All Mail.

[M-wada (m-wada)]

DUP of Bug 502370? See also Bug 511073.

[Umang Varma (umang)]

I'm really not sure which bug I'm talking about (i.e Bug 502370 or Bug 518581), but I have noticed a few things here:

0. (Yes, zero) It seems to me that TB doesn't expect mails to be marked as unread on the server when it hasn't done marked it itself. Put simply, TB is coded to believe that only it is capable of marking a mail read or unread while it is running. This is expected behavior and this is not where we want to fix this bug. (Unless you want TB to check the message ID, which may or may not be a nice thing to do. But read on, I think this issue can be fixed in a different way)

1. This can happen backwards as well. I can read the mail in All Mail and then check the mail in the other 'Labels' that mails is in and the mail will not be marked as read in these other Folders/Labels. If the initial problem is understood, this should be as well, because one of the reasons (I'm coming to the other) this problem exists is that TB doesn't recognize the Label system, and tries to apply the folder system to GMail. By this logic, the reverse issue should affect us in theory, and it does.

2. This bug is not 100% reproducible. If I read (say) 10 unread mails in my Inbox, reading the first 7 for a long time and quickly glancing over the last three, just to make them read, and immediately go to All Mail, TB checks for new mails and marks ONLY the last three as read, leaving the first seven that I took a long time to read as unread. When I have a mail in three Labels, and I read it in one, it is often marked as read in one of the other two folders while unread in the last one.

Here's my guess, for why TB behaves that way:
It does seem that when a 'folder' is not opened for a long time, the type of checking that TB does (in the background) is different from the type of checking that TB does when I open a folder (in the foreground). If a "background"/automatic check is done on the server, TB seems to check for new messages only. It doesn't check for any other changes. When it does a full check, only the changes since the last "check" seem to be recognized.

How this can be verified (over the next couple of days, I shall try these tests and post the results here):

  a. Disable all automatic and periodic checks ("Check this folder for new messages" in the folder preferences) on one folder. Read a lot of mails that are in that folder from "All Mail" and wait at least ten to fifteen minutes. Now open that folder. If this theory is correct, TB will correctly mark all read messages in that folder as read.

  b. Enable automatic checking of a folder and change settings to check every minute. If this theory is correct, this bug will be far more reproducible and should happen for almost every message. At the moment, this is reproducible but doesn't happen for all mails.

To summarize point No. 2, TB, I think, checks the server only for new mails when it does an automatic check, but when it does a proper check (when you open that folder) it looks only for changes after the last automatic check.

3. The list of read/unread messages syncs itself properly if I restart TB. Since this is the first time I'm checking in that session, I presume TB does a more thoro...

[Umang Varma (umang)]

(Forgot to add) Of course, if what I wrote above is correct, then the problem lies in TB, not in GMail.

[M-wada (m-wada)]

I could see similar phenomenon with Tb 3.0.3, with Gmail IMAP, using single IMAP client(Tb), during test for Bug 450246(tag case) and bug 544439(\Seen flag case).

1. A mail has Gmail Label of [Gmail]/All Mail, Inbox, X/A1, X/A2, X/A3, ...
   "Read status" at all folder
2. Change mail in X/A1 to Unread
   => mail in some folders was changed to Unread. (not opened yet?)
   => mail in Inbox was not changed to Unread.
   => mail in other foldrs was not changed to Unread. (already opened?)
"Work Offline, go back to Work Online, Click the folder" was required to reflect status change.
So, it looked for me next phenomenon.
  If IMAP folder is already opened, re-synchronization of flag is not executed.

Dave Carner(bug opener) and Umang:

Can you get IMAP log and check IMAP level flow?
> https://wiki.mozilla.org/MailNews:Logging
> http://www.mozilla.org/projects/nspr/reference/html/prlog.html#25328
> SET NSPR_LOG_MODULES=timestamp,imap:5 (MS Win example)

Does Gmail IMAP return new status("no \Seen flag" status or "\Seen flag" status) after status change by other for the mail?
Does Tb know "change from with \Seen flag to without \Seen flag or vice versa" of the mail in Gmail IMAP folder?
Or Tb knows it but status of MsgDB/thread pane is not refreshed?

> and immediately go to All Mail, TB checks for new mails and marks ONLY the last three as read,

Tb knows highest UID(N) which is used and next UID(N+1) for new mail. Upon new mail check, Tb probably issues "uid (N+1):* flags". If no new mail with UID grater than N, Gmail returns flag status for uid=N. If new mails for Tb exist, Tb can obtain flag status for UID from (N+1) to (N+x). I guess you are looking phenomenon at this stage.
For old mail(UID M, M is less than N), flag status change(\Seen<->No \Seen) can't be obtained unless "uid M fetch flags ..." is issue again. I guess "status of old mail is not changed" is problem around this(phenomenon I saw).

[M-wada (m-wada)]

When already accessed Inbox is clicked after flag status change at other Gmail IMAP folder, Tb 3 issued next fetch(highest UID Tb knows==11667, no new mail is added to Inbox).
> S-INBOX:SendData: 33 UID fetch 11668:* (FLAGS)
> S-INBOX:CreateNewLineFromSocket: * 4 FETCH (UID 11667 FLAGS (NonJunk \Seen))
> S-INBOX:CreateNewLineFromSocket: 33 OK Success
As Gmail IMAP returns flag data of highest UID=11667 when no new mail, Tb can know flag change of highest UID(=11667 in this case) by the fetch command. So, if last mail(highest UID), this bug's problem(\Seen flag) and Bug 450246 comment #15(tag case) doesn't occur.

[M-wada (m-wada)]

As Bug 436315 exists, I tried "Compact", but tag update at other folder was not refleted to mail in Inbox, except last mail(highest UID).
> Bug 436315 don't need to fetch 1:* FLAGS after expunge
Some enhancements like next, "Re-synchronize IMAP flag" menue, will be required?
> 324710 Consider fetching IMAP message flags in the background
Or, "MsgDB is not closed" is cause? (If closed and re-opened upon click, re-synchronization of all mails is executed by open.)

Confirming with my test result.

[Umang Varma (umang)]

> Dave Carner(bug opener) and Umang:
>
> Can you get IMAP log and check IMAP level flow?
> > https://wiki.mozilla.org/MailNews:Logging
> > http://www.mozilla.org/projects/nspr/reference/html/prlog.html#25328
> > SET NSPR_LOG_MODULES=timestamp,imap:5 (MS Win example)
>
> Does Gmail IMAP return new status("no \Seen flag" status or "\Seen flag"
> status) after status change by other for the mail?
> Does Tb know "change from with \Seen flag to without \Seen flag or vice versa"
> of the mail in Gmail IMAP folder?
> Or Tb knows it but status of MsgDB/thread pane is not refreshed?

Hi,
I tried running:

  $ NSPR_LOG_MODULES=imap:5 thunderbird

But there was way too much text being bombarded at me to be able to understand anything. Can you explain, since I haven't done this before, what exactly I'm looking for?

Also, I've noticed that the behavior is a lot more random than I thought it was. I have not been able to successfully "corner" Thunderbird and predict that if I do, XYZ, TB will do this. In my initial comment (#3), what I said sounded like the bug was predictable. It's not so.

[M-wada (m-wada)]

(In reply to comment #8)
> Also, I've noticed that the behavior is a lot more random than I thought it
> was. I have not been able to successfully "corner" Thunderbird and predict that
> if I do, XYZ, TB will do this. In my initial comment (#3), what I said sounded
> like the bug was predictable. It's not so.

Do you enable "Check for new messages upon sartup/every NN minutes"? If so, it may be by fix of bug 540214, because bug 540214 is landed on Tb 3.0.2 and MsgDB is properly closed after STAT by the fix.
If closed and re-opened(or Work Offine, back to Work Online, click the folder), fetch uid 1:* flags is probably issued and flag changes of old mails are obtained.

> But there was way too much text being bombarded at me to be able to understand
anything.

Get log with small folders, please.
(1) Create folder X/A1, X/A2, X/A3, Show "Order Received" column(UID of mail).
(2) Copy three mails to X/A1, add tag of "To Do"(blue) to three mails.
    X/A1 : change mail status
    X/A2 : opened before change at X/A1
    X/A3 : not opened before change at X/A1. opened after change at X/A1
    mail-1 : to make mail-1 last accessd mail, to keep other's Unread status.
    mail-2 : mail of non-highest UID
    mail-3 : mail of highest UID
(3) Copy mail-1(lowest uid) to X/A2, X/A3, copy mail-2(next uid) to X/A2, X/A3,
    and copy mail-3(highest uid) to X/A2, X/A3 (to keep same order).
    At all 3 folders, click mail-1(last accessed mail, to keep Unread of 2/3).
    Check all mails in all three folders, and confirm "same status/tag".
(4) Restart Tb with IMAP logging, and Click X/A2, then click X/A1
(5) At X/A1, add tag of "Important"(red) to all three mails,
    change mail-2/mail-3 to Unread, and click mail-1.
(6) Click X/A2, check thread pane(don't touch mail, not to change read/unread)
(7) Click X/A3, check thread pane(don't touch mail, not to change read/unread)
What status/tag(color) is seen for mail-2/3 in X/A2 and X/A3?
(8) Work Offline, go back to Work Online
(9) Click X/A2, check thread pane(don't touch mail, not to change read/unread)
(10) Click X/A3, check thread pane(don't touch mail, not to change read/unread)
What status/tag(color) is seen for mail-2/3 in X/A2 and X/A3?
(8) Terminate Tb, keep backup of log.
Note: above is similar procedure I used in test of Bug 450246 for tag.

As test is relevant to X/A1, X/A2, X/A3 only, irrelevant logs can easily be removed by text editor. To reduce irrelevant logs since first, I recommend to disable "Check for new messages upon sartup/every NN minutes" when getting log.

[M-wada (m-wada)]

Umang, no need to execute test of comment #9.

I could observe that \Seen/no \Seen and addition/removal tag at other folder is immediately reflected to any other folder with Tb 3.0.3 like before.
Reflection to Inbox was after several click of Inbox. It's probably by "cached MsgDb after a while"(still cached, then "uid 1:* flags" is not issued).

When I tested comment #6(and bug 450246 #13 to #15), I saw "too much connections" from Gmail IMAP because I used both Sm2 and Tb3 for same Gmail IMAP access with "max cached connections=5"(Sm2: 2 Gmail IMAP, Tb: 1 Gmail IMAP) and I repeated termination/restart of Tb for testing. Such error may produce phenomeon of "unclosed MsgDB or close failure of MsgDB".
I'll continue to check it in bug 450246.

[M-wada (m-wada)]

I could observe "UID fetch 1:* (FLAGS)" upon click of Inbox which was opened in the past.

> Folder is switched at folder pane multiple times.
> Before next, S-INBOX:SendData: 1xx select "S-A-00/A-01" occurred.
>
> At this step, Inbox.msf is opened(checked by MSGDB:5)
> Click Inbox.
> S-A-00/A-03:SendData: 127 select "INBOX"
> S-INBOX:CreateNewLineFromSocket: * FLAGS (\Answered \Flagged \Draft \Deleted \Seen)
> S-INBOX:CreateNewLineFromSocket: * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen \*)]
> S-INBOX:CreateNewLineFromSocket: * OK [UIDVALIDITY 602312987]
> S-INBOX:CreateNewLineFromSocket: * 4 EXISTS
> S-INBOX:CreateNewLineFromSocket: * 0 RECENT
> S-INBOX:CreateNewLineFromSocket: * OK [UIDNEXT 11668]
> S-INBOX:CreateNewLineFromSocket: 127 OK [READ-WRITE] INBOX selected. (Success)
> S-INBOX:SendData: 128 getquotaroot "INBOX"
> S-INBOX:CreateNewLineFromSocket: * QUOTAROOT "INBOX" ""
> S-INBOX:CreateNewLineFromSocket: * QUOTA "" (STORAGE 11514 7609372)
> S-INBOX:CreateNewLineFromSocket: 128 OK Success
> S-INBOX:SendData: 129 UID fetch 1:* (FLAGS)

What I saw in comment #6 was possibly phenomenon like next;
- Until "select Inbox" happen again, UID fetch 1:* (FLAGS) is not issued again.
- If connection error occurs on connection for S-INBOX, folder switch by
  "select other-folder" at the connection doesn't occur not so easily.
- "MsgDB is not closed due to soemthing" may produce similar issue,
  because folder open is not invoked.

[M-wada (m-wada)]

Umang, can you get log while you are using Tb with next parameter?
> SET NSPR_LOG_MODULES=timestamp,MSGDB:5 (MS Win example)
Run one to two hours with logging, and check whether "number of opened .msf" increases without close, or not, please. (log of "increase as if infinite" was attached to a bug, although it looked local folder/rss/news case.)

[Umang Varma (umang)]

Again, I'm not following this very well, but if I do this (On Linux):
  $ NSPR_LOG_MODULES=timestamp,MSGDB:5 thunderbird
Then I'm getting a lot of output again. However, it has the names of many of my folders (which I would not like to post on publicly accessible page like this).

Should I just do a
  $ NSPR_LOG_MODULES=timestamp,MSGDB:5 thunderbird > tb.log
and send the `tb.log` file to you by email?

[Umang Varma (umang)]

Forgot to ask, when I check all these logs, do I enable or disable "Check for new messages every 5 minutes"? (It's enabled right now)

[M-wada (m-wada)]

(In reply to comment #13)
> Should I just do a (snip) and send the `tb.log` file to you by email?

There is no need of log data itself. Your check result of "number of opened .msf at each MSGDB:5 logging(looks periodical) increases as if infinetely or not" and of "close of .msf/MsgDB is logged as expected or not" is sufficient.

(In reply to comment #14)
> do I enable or disable "Check for new messages every 5 minutes"? (It's enabled right now)

If you use Tb 3.0.2 or later, bug 540214 is fixed. So, I think check with enabled is better initially, if enabled is your default setting for daily use, because folder access/MsgDB open doen't happen unless you manually click folder if disabled.
If something wrong around MsgDB open/close will be found, test with disabled will be required for diagnosis, to check whether other component such as Gloda, auto-sync, is culprit or not.

[M-wada (m-wada)]

Test result of comment #11 is with "check for ...startup/every 1 minute" enabled, with "max number of cached connections=1" to force folder switch at connection upon each folder click.
Umang, can you check with "max number of cached connections=1"

[M-wada (m-wada)]

Phenomenon I saw with Tb 3.0 in bug 450246 comment #13 to #15 was same problem as this bug.

[Umang Varma (umang)]

This is the kind of output I get:

Many lines like this:

> 2010-03-09 12:58:38.440820 UTC - -1215604896[b7611060]: /home/umang/.thunderbird/qu5s4b6i.default/ImapMail/imap.googlemail.com/[Gmail].sbd/All Mail.msf - 3 hdrs in use

then "closing database" after every once in a while:

> 2010-03-09 12:59:44.252318 UTC - -1215604896[b7611060]: closing database /home/umang/.thunderbird/qu5s4b6i.default/ImapMail/imap.googlemail.com/[Gmail].sbd/All Mail.msf

and some like this:

2010-03-09 13:02:01.135137 UTC - -1215604896[b7611060]: 5 open DB's

then I see more of the first kind of output:

> 2010-03-09 13:03:27.404664 UTC - -1215604896[b7611060]: /home/umang/.thunderbird/qu5s4b6i.default/ImapMail/imap.googlemail.com/INBOX.msf - 166 hdrs in use

What is the number I am looking for? I've not exactly understood what I have to do. I can't find "number of opened .msf" anywhere.

[M-wada (m-wada)]

(In reply to comment #18)
> What is the number I am looking for?

I wanted to say NNN in "NNN open DB's" of next log by the "number".
> and some like this:
> 2010-03-09 13:02:01.135137 UTC - -1215604896[b7611060]: 5 open DB's
If problem of "MsgDB is never closed" occurs, the NNN increases as if infinitely. So, I'm confident that such problem doesn't exist in your environment by your answer. Thanks for your log analysis.

Note: "PPP hdrs in use" is probably similar to "PPP mails are held in the folder".

[Bienvenu (bienvenu)]

hdrs in use refers to how many nsMsgHdr objects are in memory for that folder. It helps us know how much more memory is being used for that folder, and also what the folder is being used for (things like views and gloda indexing create nsMsgHdrs when they use a db; get/setting a property on a folder in the db doesn't cause any headers to get instantiated)

[Umang Varma (umang)]

Any progress? Is there any way I can provide more information?

[Michał Gołębiowski-Owczarek (mgol)]

User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.6) Gecko/20100628 Ubuntu/10.04 (lucid) Firefox/3.6.6
Build Identifier: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.10) Gecko/20100528 Lightning/1.0b1 Thunderbird/3.0.5

In Thunderbird only one of newly archived and marked as read before is marked as read in All Mail directory.

Reproducible: Always

Steps to Reproduce:
1. Wait for a few messages to arrive in Inbox, thus being marked as unread.
2. Mark all of them as read, then archive them.
3. Click on All Mail folder.
Actual Results:
Only one of recently archived messages is marked as read (usually the newest one), the rest must be marked as read again.

Expected Results:
All of recently marked as read in Inbox and archived mails should be marked as read in All Mail folder, too.

[Michał Gołębiowski-Owczarek (mgol)]

Binary package hint: thunderbird

Steps to reproduce:
1) Wait for a few messages to arrive in Inbox, thus being marked as unread.
2) Mark all of them as read, then archive them.
3) Click on All Mail folder.

Actual Results:
Only one of recently archived messages is marked as read (usually the newest one), the rest must be marked as read again.

Expected Results:
All of recently marked as read in Inbox and archived mails should be marked as read in All Mail folder, too.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: thunderbird 3.0.5+build2+nobinonly-0ubuntu0.10.04.1
ProcVersionSignature: Ubuntu 2.6.32-23.37-generic 2.6.32.15+drm33.5
Uname: Linux 2.6.32-23-generic x86_64
NonfreeKernelModules: wl
Architecture: amd64
Date: Fri Jul 9 02:22:30 2010
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
SourcePackage: thunderbird

[Ludovic-mozillamessaging (ludovic-mozillamessaging)]

*** This bug has been marked as a duplicate of bug 518581 ***

[Ludovic-mozillamessaging (ludovic-mozillamessaging)]

*** Bug 577586 has been marked as a duplicate of this bug. ***

[Micah Gersten (micahg)]

Marking triaged as we have an upstream bug.

[Ludovic-mozillamessaging (ludovic-mozillamessaging)]

*** Bug 650103 has been marked as a duplicate of this bug. ***

[Umang Varma (umang)]

I am still affected on TB 5.

[John Park (johndouglaspark)]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.835.202 Safari/535.1

Steps to reproduce:

When reading an email or emails on another device/location, those emails are marked as read on that device and the IMAP server. I am using gmail & an IMAP connection on both Thunderbird and the other device.

Actual results:

When Thunderbird is left open, it sometimes takes 2-3 hours or even longer for Thunderbird to mark those emails as read. I have tried clicking on the "Get Mail" button to refresh the IMAP folder but TB still doesn't mark those emails as read. I have checked to see whether those emails are flagged as read or unread but using the gmail web client and they are indeed being marked as read but Thunderbird will not refresh their read/unread status.

Expected results:

Each time Thunderbird attempts to get mail it should check the status of read and unread emails from the IMAP server and mark those emails as read if applicable. The only way to force TB to refresh the read/unread status is to completely close TB and re-open which is obviously very annoying.

[M-wada-7 (m-wada-7)]

Do you enable IDLE command use?
If yes, same problem as bug 512745.
  bug 512745 :
    Gmail Label of A and B is added to a mail, and has flag \Seen(or no \Seen).
    The mail is shown in both Gmail IMAP folder named A and B,
    and Tb knows flag \Seen(or \no \Seen) of both mails.
    Mail in Gmail IMAP folder named A is changed to no \Seen(or has flag \Seen).
    Mail in Gmail IMAP folder named B is also changed to no \Seen(or has flag
    \Seen) at Gmail IMAP server, because both are copy of same mail each other.
    This flag status change at Gmail IMAP folder named B is not notified to
    IMAP client like Tb via IDLE.
  Your report :
    A mail in a Gmail IMAP folder has flag \Seen(or no \Seen).
    Tb1(or Smart Phone etc.) and Tb2 knows flag \Seen(or \no \Seen) of the mail.
    The mail is changed to no \Seen(or has flag \Seen) by Tb1(or Smart Phone).
    The flag status change at Gmail IMAP folder is not notified to Tb2 via IDLE.
These are same phenomenon.

> I have checked to see whether those emails are flagged as read or unread
> but using the gmail web client and they are indeed being marked as read
> but Thunderbird will not refresh their read/unread status.

If the mail is "a mail in a conversation of Gmail which contains multiple mails", phenomenon of bug 478996 occurs. Don't use "read/unread status of a conversation at Gmail Web interface" as evidence of "read status or unread status of a mail", unless it's sure that the conversation of Gmail has single mail only or that all mails in a conversation of Gmail has same status of read or unread.

See also bug 518581, if Gmail IMAP.

> Expected results:
> Each time Thunderbird attempts to get mail it should check the status of read
> and unread emails from the IMAP server and mark those emails as read if applicable.
> The only way to force TB to refresh the read/unread status is
> to completely close TB and re-open which is obviously very annoying.

To avoid excess network traffic due to "fetch of flags of all mails in IMAP folder upon each folder access",
(imagine 1,000,000 mails in a folder * 10,000 folders, new mail check of all mails folders each one minute)
Tb currently relies on "flag status change notification from server via IDLE", but many IMAP servers including Gmail IMAP unfortunately don't look to do so.
Following is a possible workaround at Tb in your case, if your any IMAP folder is never huge.
  - Disable IDLE command use
    Max chached connections = 1
    (I think Smart Phone's setting is similar to: no IDLE, max=0)
    (Tb doesn't permit max=0. Always uses default of 5 if max=0 is specified)
  - When you want to know flag status change by other IMAP cilent,
    or when you want to know flag status change propagation to
    same mail in other Gmail IMAP folder,
    click different folder, then click needed folder at folder pane of Tb.
Can you see flag status change by other IMAP client like Smart Phone, using Tb without restart?

[John Park (johndouglaspark)]

I tried disabling IDLE and setting max cached connections to 1 but there appeared to be no change to the issue.

I also changed max cached connections to 0 and still no change.

I tried selecting multiple folders (Sent Items, Drafts etc) to try and force TB to refresh the read flags within the Inbox folder but no joy.

Interestingly, if I read an email in TB, that email is almost instantly flagged as read on my iPhone.

There is still a delay of many hours if I read an email on my iPhone or web client to be marked as read in TB.

[John Park (johndouglaspark)]

Just noticed your notes regarding max cached connections = 0 not being a valid option. Changed back to 1 and still no change in result.

The only way I can get TB to check for read/unread status is to exit TB and relaunch.

[M-wada-7 (m-wada-7)]

(In reply to John Park from comment #2)
> I tried disabling IDLE and setting max cached connections to 1 but there
> appeared to be no change to the issue.

Did you restart Tb after the setting change?
Effect by the setting change is not immediate, so restart of Tb is needed to see effective or not.

(In reply to John Park from comment #3)
> Just noticed your notes regarding max cached connections = 0 not being a
> valid option. Changed back to 1 and still no change in result.

Read next in my comment, please.
> (Tb doesn't permit max=0. Always uses default of 5 if max=0 is specified)

[John Park (johndouglaspark)]

I've restarted TB and it appears the emails are now being marked as read.

Thank you for your help!

Is this an issue with TB or google?

[Mozilla-ex (mozilla-ex)]

It's an issue with how google handles synchronizing state between multiple connections. It's up to the server how frequently it does that synchronization...

[M-wada-7 (m-wada-7)]

(In reply to John Park from comment #2)
> Interestingly, if I read an email in TB, that email is almost instantly
> flagged as read on my iPhone.

Is this true for a mail in other than Inbox?
  At Tb, move(not copy) some(not one) mails from Inbox to other than Inbox
    (call Test folder, not subfoler of Inbox)
  At iPhone, see the mails in folder named Test.
  At iPhone, see other folder than the Test folder, such as Inbox.
  At Tb, show "Order Received" column of Test folder.
  At Tb, change Read/Unread status of a mail in Test folder
         which has smallest "Order Received" column value.

[M-wada-7 (m-wada-7)]

Per folder setting like next may be a solution.
  [ ? ] When automatic new mail check is invoked for this folder,
        always checks flag/keyword status of all mails in this folder.

[M-wada-7 (m-wada-7)]

*** Bug 568775 has been marked as a duplicate of this bug. ***

[Demaine Benoît-Pierre (doublehp)]

Tagging the old report as dup of the new one is the best way to make devs think the bug is recent (and irrelevant).

Yes, an other TB3 profile on the same host.

But bug can also be reproduced with any other Imap client, like my mobile phone: just after reading an email on Gmail, I can sync my phone, and it will show mail as read; but clicking GetMail on TB3 will now ... untill TB restart. This is making 568775#c6 irrelevant.

[Demaine Benoît-Pierre (doublehp)]

An other point against TB, prooving that it's not Google's fault: bug appeared during the week after migration from TB2 to TB3 ... it would be a very strange hasard if Google changed their server conf just that week ...

[Ww3ib0sg9w-tim-8aw3u04umo (ww3ib0sg9w-tim-8aw3u04umo)]

I am experiencing this on TB 11.0.1, OSX 10.7.3.

Maybe something in my settings, though, because it seems to be occurring on one account in particular (amongst several gmail imap accounts).

I delete from Inbox ('Remove It Immediately') and it remains unread in All Mail, and shows up for an extended period of time in the new mail notifier.

[M-wada-7 (m-wada-7)]

FYI.
Another workaround, if "max cached connections=1" is not practical in your environment, and if you are eager to know flag/keyword change of existent mails by other mail client even though your IMAP server doesn't notify flag/keyword change to Tb via IDLE :
  Go "Work Offline mode", then go back to "Work Online mode",
  and, if folder you want to know flag/keyword change by other is FolderX,
  click an account or other folder, then click FolderX.

[David Lechner (dlech)]

We could take advantage of the X-GM-LABELS and X-GM-MSGID attributes to fix this bug.

Algorithm would be: If message flags are updated and message has X-GM-LABELS, then iterate labels and find messages with matching X-GM-MSGID and update the flags for those messages.

[King-y (king-y)]

Still experiencing this with 17.0.5, does any further information need to be provided?

[M-wada-7 (m-wada-7)]

FYI.
Gmail IMAP started to support CONDSTORE from this year, so problem in Tb's CONDSTORE support is perhaps exposed to all Gmail users(see bug 912216, bug 885220).
IIUC, workaround of "Going offline then online" is affected by the problem and the workaround perhaps doesn't work as expected(see bug 885220 cmment #76).
Disable Tb's CONDSTORE support, please.
  mail.server.default.use_condstore = false (setting for all IMAP accounts)
  mail.server.server#.use_condstore = false (per account setting)

[M-wada-7 (m-wada-7)]

FYI.
If server supports CONDSTORE, and if CHANGEDSINCE of CONDSTORE is used by Tb upon new mail check, problem of this bug can be resolved.
1. Upon first open(select at cached connection) of Mbox after restart.
   select "Mbox" (CONDSTORE)
   * OK [HIGHESTMODSEQ 120418]
   UID fetch 1:* (FLAGS)
2. Upon second or later open(select at cached connection) of Mbox after restart of Tb.
   select "Mbox" (CONDSTORE)
   * OK [HIGHESTMODSEQ 120418]
   UID fetch 1:* (FLAGS) (CHANGEDSINCE 120418)
3. Upon each new mail check at cached connection where Mbox is already selected.
   UID fetch 1:* (FLAGS) (CHANGEDSINCE 120418)
    * 836 FETCH (UID 3292 MODSEQ (120420) FLAGS (\Seen))

[M-wada-7 (m-wada-7)]

*** Bug 512745 has been marked as a duplicate of this bug. ***

[M-wada-7 (m-wada-7)]

"IDLE does not send unsolicited responses for flag changes" is not Gmail IMAP only issue.
Issue of "Tb is not torelant with 'IDLE does not send unsolicited responses for flag changes'" is not Gmail IMAP only issue.
Because bug 693204 was processed after analysis of many bugs, that bug is crispy than other bugs.
So, duping this bug to bug 693204.
If duping is wrong, re-open this bug, please.

*** This bug has been marked as a duplicate of bug 693204 ***

[M-wada-7 (m-wada-7)]

*** Bug 518581 has been marked as a duplicate of this bug. ***

[M-wada-7 (m-wada-7)]

*** Bug 594106 has been marked as a duplicate of this bug. ***

[M-wada-7 (m-wada-7)]

*** Bug 672585 has been marked as a duplicate of this bug. ***

[M-wada-7 (m-wada-7)]

*** Bug 573985 has been marked as a duplicate of this bug. ***

[M-wada-7 (m-wada-7)]

*** Bug 606395 has been marked as a duplicate of this bug. ***

[M-wada-7 (m-wada-7)]

*** Bug 577586 has been marked as a duplicate of this bug. ***

[M-wada-7 (m-wada-7)]

*** Bug 650103 has been marked as a duplicate of this bug. ***

[William Paul Liggett (junktext-0)]

Thanks for the information, WADA. The easy way to disable the CONDSTORE as suggested on Comment #13 is by doing this:

1. [On Linux] Edit -> Preferences // [On Windows] Tools -> Options...
2. Advanced -> Config Editor...
3. Search for "CONDSTORE" (with no quotes, and it shouldn't matter if it's lowercase).
4. Double-click on "mail.server.default.use_condstore" to change the 'Value' to "false".
5. Close the Config Editor (the search window).
6. Restart Thunderbird.

I hope this works, as I am just trying this fix out myself.

(See my old bug comment here: https://bugzilla.mozilla.org/show_bug.cgi?id=594106#c43)

[Nyet (nyet)]

Again, disabling CONDSTORE does not always help. There should probably be two separate bugs, one for CONDSTORE related, one for not related to CONDSTORE.. a bunch of different bugs all got rolled into one bug, so if somebody fixes CONDSTORE, hopefully they will not close this :/

[M-wada-7 (m-wada-7)]

(In reply to Nye Liu from comment #24)
Nye Liu, please don't confuse following.
 (a) problem of this bug(notification of flag status change)
 (b) problems due to CONDSTORE support in Tb
 (c) Tb's behavior in CONDSTORE support which may affect on workaround of this bug;
       - Going Work Offline, going back Work Online, then re-open folder by folder click
       - max cached connection=1, re-open folder by folder click
 (d) A possible solution of this bug by utilizing CONDSTORE (by correctly using)

"Disabling CONDSTORE support of Tb" in this bug is only for (c), even though the "disabling CONDSTORE support of Tb" is automatically applied to (b) always.
Purpose of workaround of this bug is to force Tb to issue "uid fetch 1:* flags" upon "folder click" step in workaround.
However, as I already pointed in comment #13, Tb may issue "UID fetch 1:* (FLAGS) (CHANGEDSINCE mod-seq-value)" or "uid fetch HighestUID:* (FLAGS)" upon the folder click in the workaround when CONDSTORE support of Tb is enabled. And, it may be different between Inbox and other Mbox. It may depend on timing of events during workaroud execution. Inconcistency of phenomenon during workaround execution surely produces user's confusions.
Purpose of "Disabling CONDSTORE support of Tb" in this bug is:
  Surely force "uid fetch 1:* flags" upon "folder click" step of workaround.
  Avoid user's confusions during workaround execution.

Because problem of this bug itslef is absolutely irrelevant to CONDSTORE support by Tb, even if CONDSTORE extension can be utilized to resolve problem of this bug, I'm sure that developers never confuse this bug with CONDSTORE support related problem.

[M-wada-7 (m-wada-7)]

(In reply to Nye Liu from comment #24)
> There should probably be two separate bugs, one for CONDSTORE related, one for not related to CONDSTORE..
> a bunch of different bugs all got rolled into one bug, (snip)

Nye Liu, have you actually read and understood all bugs which is duped to this bug by me?
Which phenomenon in which dup'ed bug is not problem due to "flag status change is not notified from IMAP server via IDLE"?

Do you actually understand problems by CONDSTORE support of Tb or problems in Tb around CONDSTORE?
Have you read and understand Bug 885220　which I already pointed in comment #13, which is for problem occurs only when CONDSTORE support of Tb is enabled?
Do you understand a change in the past which is one of causes of Bug 885220?
Have you read and understand bug 885220 comment #88?
Do you understand why the "change in the past" was backed out?
Do you understand what problem may occur again by the backuot?
Do you understand what problem occurred by the backout?
Have you read and understand bug 912216 which I already pointed in comment #13?

These problem relevant to "CONDSTORE extention and Tb's CONDSTORE support" and problem relevant to "flag status change is not notified from IMAP server via IDLE" are clearly isolated already.

[M-wada-7 (m-wada-7)]

Quick summary of this bug, to avoid confusions by some peoples, or to avoid irrelevant comments due to confusion with problems in Tb's CONDSTORE support.

-Problem of this bug;
  - Because server doesn't notify flag status change via IDLE,
    Tb can't know flag status change of mails at server Mbox via IDLE.
  - Tb's expectaion of "server notifies flag status change via IDLE" is never
    RFC violation, because "flag status change notification via IDLE" is SHOULD.
  - Server's behavior is never RFC violation, because "flag status change notification
    via IDLE" is SHOULD, is never MUST.
  - Unfortunately, some major IMAP servers don't notify flag status change via IDLE,
    and Gmail IMAP is an example of such server.
- Problems in current "Tb's CONDSTORE suport" is irrelevant to above problem of this bug
  which is produced by IDLE command use.

Please never confuse following.

- If CODNSTORE extention is well utilized, problem of this bug can be resolved by simple
  method.

- Current Tb's CONDSTORE support has some problems, so "disabling Tb's CONDSTORE"
  is better. Rather, Tb user MUST disable it until problems will be resolved,
  if Tb user uses IMAP server who supports CONDSTORE extension.

- Gmail started to support CONDSTORE extesion this year,
  and Tb's default is mail.server.default.use_condstore=true.
  So problem in "Tb's CONDSTORE support" is exposed to Tb/Gmail IMAP user from this year.
  Because number of Tb+Gmail IMAP user is not so small,
  default of mail.server.default.use_condstore should be changed to false.

  "dup'ed bugs to this bug which is relevant to Gmail IMAP" was opened before Gmail
  started to support CONDSTORE extension. How can problem reported to these dup'ed bugs
  be relivant to Tb's CONDSTORE support and/or CONDSTORE extesion?

- Because wrokaround of this bug is affected by CONDSTORE extension and Tb's current
  CONDSTORE support, Tb user who experienced problem of this bug should disable
  Tb's CONDSTORE support, if Tb user uses IMAP server which supports CONDSTORE,
  in ordre to make the workaround usable,
  in order to avoid confusion produced by problem in Tb's CONDSTORE support.

- Because "disabling Tb's CONDSTORE support" does do nothing
  if IMAP server doesn't support CONDSTORE or server is not IMAP server,
  Tb user can disable Tb's CONDSTORE support freely, safely, any time,
  regardless of "used IMAP server supports CONDSTORE or not",
  regardless of "IMAP server is used or not".

[Brian Norris (computersforpeace)]

Reassigned to the duplicate Mozilla bug. Unfortunately, this resets the Lanuchpad ticket to "Unkwown" status and "Unknown" importance instead of "Confirmed" status and "Medium" importance. Unfortunately, I can't edit these fields...

[Nyet (nyet)]

(In reply to WADA from comment #27)
> - Tb's expectaion of "server notifies flag status change via IDLE" is never
> RFC violation, because "flag status change notification via IDLE" is
> SHOULD.

I'd say that is only true if "flag status change notification via IDLE" is MUST.

Seems to me that unless the server MUST provide flag status changes via IDLE, TB cannot rely on the server to provide flag status changes.

[Julian J. M. (julianjm)]

Years ago it worked... I guess it was from the Thunderbird 2-3 times.

Are we sure the problem is related to IDLE flag updates? Why do other IMAP clients detect deletions almost instantaneously?

I have Thunderbird 24.1.0 and K9Mail on android both connected to the same IMAP account on Zimbra 8.
If I delete a message on Thunderbird I see the action on the phone in 2-3 seconds.
If I delete a message on the phone, i never see the message disappear on Thunderbird.

Going to offline mode and back doesn't seem to do it for me.

[M-wada-7 (m-wada-7)]

*** Bug 712595 has been marked as a duplicate of this bug. ***

[LonelyPixel (nospam-unclassified)]

Guess what, this isn't even limited to Gmail. It happens on my own IMAP server as well. It's running Dovecot on Linux. See bug 712595. Offline and online mode does not help. Changing folders does not work. Only restarting Thunderbird helps for sure. Until then, I regularly see unread or read messages in my inbox at home that I've already read and/or deleted through webmail or smartphone. I also have the impression that it's not a long delay, but instead it's a now or never. Disabling IDLE is definitely no option because that's the future (well, present already) and what everybody else uses successfully. I have the impression that there's some race condition inside Thunderbird that nobody will ever be able to find without rewriting the whole thing.

[M-wada-7 (m-wada-7)]

(In reply to Yves Goergen from comment #31)
> Guess what, this isn't even limited to Gmail. (snip)

What is reason to add new "merely 'me too' comment without required data for problem analysis" to this bug, even after reading my bug 712595 comment #24?
There is no need of such "guess" in this bug. In this "Bug report at B.M.O for developers to resolve Tb's actual bug(flaw in Tb's code)", "Why problem occurs", "What hapens" etc. is pretty clear already.
Please note that here is never support forum nor Help Center to saying complaint to Tb developers. Here is B.M.O for helping developers.
All "comuication between IMAP cliet(Thunderbird) and IMAP server" is done by "IMAP command and responce". How can software developers know "what actually happens in your environment" without required data?

[Daze (daze)]

4 x windows xp machines running thunderbird 24.1.1
2 x iphones (ios 6 and 7)

All sharing 4 different mailboxes.

Up until recently we used ms but changed due to roaming issues without exchange.

Our email is hosted by gmail.

All email was replicated instantly using outlook so gmail is not the problem.

Results are hit and miss, some machines update immediately, others will not update unless tb is restarted, offline/online / different folder / manually checking for mail does not solve.

Settings on all machines / users / mailboxes are mirrored. IDLE is enabled accross but i have disabled and makes no difference. I have disabled firewalls incase it was stopping the server updating but still nothing.

[Demaine Benoît-Pierre (doublehp)]

(In reply to WADA from comment #32)
> All "comuication between IMAP cliet(Thunderbird) and IMAP server" is done by
> "IMAP command and responce". How can software developers know "what actually
> happens in your environment" without required data?

Any developper can repro the bug easily: create a Gmail account (they are now free, and don't need sponsorship anymore; beta have been close, and official product have been launched years ago), configure two different Thunderbird profiles to access the same Gmail account, send one email from one box to the other one, read the mail on both, mark the mail as unread on one, and click GET MAIL on the other one, you will see that email is marked unread only after restarting "the other one".

You don't need huge amount of data from users, strace or backtraces. Any dev can produce desired amount of logs for free, any time, and from any country (yes ... I had country specific issues with Google in the past - this bug is NOT one of them). To dig deeply into client-server communication, tcpdump often helps.

Thunderbird is a free software. And dev always remind users that they are volunteers. But the products of the Mozilla fundations have a reputation to defend; they slowly became popular, and are now used by many people ... not only geeks in the cave using Linux for religious reasons; Thunderbird is used by people why have serious work, and who consider Thunderbird as a work tool. This means ... the tool must meet minimal requirements, and must work at least, as good as other concurrent products. And since mid 2013, thunderbird have serious issues. It does not sync properly messages status, and does not work at all anymore with Google Calendar (yes I know, it's not Mozilla Fundation who was responsible for Google Calendar Exchange server shut down; but the moz team could have worked on making TB compatible with an other protocol). Now, TB have become a poor mail reader not even able to track a READ STATUS, I can not use PGP encryption at all (since TB3, or maybe even TB2, I forgot), and it can not let me use my calendar any more. I wonder why I have not yet changed MUA.

Dev always claim that ... they work for free, and are free to choose what to work on, which snipset of code they want to have fun with. By letting users down on basic features, users will get bored of the product. This bug is one of the few that I consider critical and urgent. Last week I was trying to configure an old Nokia Symbian S60 for Gmail sync; in my research, I have found several articles saying that Gmail closed the Exchange access, and that Mail for Exchange for Symbian is now deprecated because of Google choices; the articles was also doing an overview of other affected products; MFE@Symbian was considered as a small loss, because Nokia themselves left down Symbian, and they officially stopped supporting Symbian S60 phones. The article also mentionned that Thunderbird was also affected, and that is was a shame for such a populat product to not publish a serious work around within days. A WA should have been proposed in days; it's now one year, and TB still does not compensate the loss; any mobile phone (included Linux based And...

[M-wada-7 (m-wada-7)]

(In reply to delid4ve from comment #33)
> (snip) offline/online / different folder / manually checking for mail does not solve. (snip)
> IDLE is enabled accross, but i have disabled and makes no difference.

It's pretty simple. Merely one of next,
  (i) Your problem is not this bug. Your problem is absolutely different problem from this bug.
  (ii) Your problem is this bug, but your operation is wrong.
because this bug is for following problem(and only for following problem).
  Server doesn't notify flag change via IDLE,
  so Tb can't know flag change status of already fetched mail via IDLE.

Example of (i) :
- If CONDSTORE is used, UID fetch 1:* (FLAGS) (CHANGEDSINCE ...), UID fetch Highest:*
  (CHANGEDSINCE ...) is used by Tb. This may produce problems in getting flag status of already
  fetched mail.
  Read bug 885220 for IMAP flow for this, please.
  This kind of issue is one of reasons why bug 912216 was opened.
  Note: Both bugs are alreeady pointed in this bug.
Example of (ii) :
  - IDLE command is issued after end of IMAP command execution at cached connection.
    "Use IDLE command" setting is for "issue IDLE or not, at end of IMAP command execution".
    "Use IDLE command" setting is never for "terminate current IDLE status at a cached connection".
  - Upon periodical new mail check, if an Mbox is already selected at a cached connection.
    Tb uses "uid fetch KnownHighestUID:* Flags".
    Inbox is selected at a cached connection always, except when max cached connection=1.
  Even though these, even though an Mbox is still IDLE status at a cached connection after
  disabling "Use IDLE command" setting, or even though an Mbox is still selected at a cached
  connection after some actions, write useless comment to a bug.
  Note:
  "go Work Offline" step of operations which is written in this bug is for;
     Force close of all cached connections, in order to force Tb to do "open connection, login,
     select Inbox(and other Mbox), uid 1:* fetch (flags)" again after "go Work Online".
  "disabling CONDSTORE support" is for;
     Force Tb to use "uid 1:* fetch (flags)" instead of "uid fetch 1:* (flags) (CHANGEDSINCE ...)"
     even when server supports CONDSTORE extension.

[Nyet (nyet)]

Would it be possible to at least add some sort of menu option next to "work offline" ... like "force sync" or something?

[William Paul Liggett (junktext-0)]

*(In reply to plop from comment #34)

Yeah, plop, that's why I switched to the Evolution mail program instead. It seems to work better with Gmail than Thunderbird. I don't wish Thunderbird to die out or anything, but I probably won't go back any time soon.

Although, I am slowly getting away from using Gmail as my main provider, since I do not really care for people having easy access to see all of my e-mail, not that I have anything to hide, but on principle. I have just been using my website's e-mail service instead.

[Miguel Rubio-Roy (mrubioroy)]

(In reply to WADA from comment #27)
> -Problem of this bug;
> - Because server doesn't notify flag status change via IDLE,
> Tb can't know flag status change of mails at server Mbox via IDLE.
> - Tb's expectaion of "server notifies flag status change via IDLE" is never
> RFC violation, because "flag status change notification via IDLE" is
> SHOULD.
> - Server's behavior is never RFC violation, because "flag status change
> notification
> via IDLE" is SHOULD, is never MUST.
> - Unfortunately, some major IMAP servers don't notify flag status change
> via IDLE,
> and Gmail IMAP is an example of such server.

Dear WADA,

How do other email clients deal with this situation on the commonly huge IMAP folders of Gmail ?

I think this is a rather critical bug of Thunderbird. As you say, "some major IMAP servers don't notify flag status change via IDLE", and Tb should be able to deal with this situation OOB. If I understand correctly, the workaround only works well with small IMAP folders.

Are there plans to fix this 2.5 years old bug ?

Regards,

[Richard-munitz (richard-munitz)]

Yes, I too have become exceedingly frustrated with this problem to the point of considering a move to another email client. Given that a very painful manual workaround is available and has been for some time, it seems criminal that these workaround steps haven't been automated within Thunderbird. Would it really be too resource intensive to simply perform these additional steps each time Thunderbird checks for mail updates? Maybe that is what every other email client does and that is why they do not have this problem. Thank you.

[M-wada-7 (m-wada-7)]

(In reply to Nye Liu from comment #36)
> Would it be possible to at least add some sort of menu option next to "work offline" ... like "force sync" or something?

A possible way.
- Install "Custom Buttons" addon, add your own Toolbar Button labeled "force sync".
- Put small JavaScript code like "click Work Offline, reply if required, wait for a while, click Work Online, reply if required".
  /*CODE*/
  // var MenuId="goOfflineMenuitem" ; // Firefox
      var MenuId="goOfflineMenuItem" ; // Thunderbird
      var Menu=document.getElementById(MenuId) ;
      var Status=( "true" == Menu.getAttribute("checked") ) ; // atribute value is String
      var OnCommand=Menu.getAttribute("oncommand") ;
      // oncommand="MailOfflineMgr.toggleOfflineStatus();"
      if( false == Status )
      {
          // go Work Offline
          eval(OnCommand);
          // do reply to dialog for download now,
          // go back Work Online, and do reply to dialog for send unsent messages
          // setTimeout(GoBackOnline,3*1000) ;
       }
      else if( true == Status )
      {
         // go back Work Online
         eval(OnCommand);
         // and do reply to dialog for send unsent messages
      }

This version doesn't do automatic reply to confirmtion dialog, doesn't do Toogle Back to Work Online mode.
So, you have to do: 1. Click the button, 2. No to dialog(download now), 3. click button, 4. Cancel to dialog(send unsent mail).
This is merely an example of your own pretty small Addon by your own ToolBar button.
id of menuitem, toolbarbutton , attributes of them etc. can be pretty easily known using DOM Inspector.

[M-wada-7 (m-wada-7)]

(In reply to Miguel from comment #38)
(In reply to Rich from comment #39)

If this bug is so critical for you and so important for you, why you never do one of next yet?
  (a) Use other than Thunderbird as your mailer which fully fulfills your pretty important requirement(s).
  (b) Use other sophisticated and clever IMAP server who supports feature of SHOULD even if the feature is not MUST in RFC.
Why you never ask your IMAP server to support feature of SHOULD even if the feature is not MUST in RFC?
Even if SHOULD != MUST in RFC, why expecting feature of SHOULD can be a critical bug of a software?
SHOULD is sufficiently strong even if it's not MUST and SHOULD is weaker than MUST, isn't it?

[LonelyPixel (nospam-unclassified)]

(In reply to WADA from comment #40)
> - Put small JavaScript code like "click Work Offline, reply if required,
> wait for a while, click Work Online, reply if required".

In case you're not aware of it anymore, going offline and back online does not resolve the issue. At least for me. The only thing that does help is restarting Thunderbird, which I have to doo every few days when I deleted something either in the web mailer or my smartphone (K9 mail), with both work fine the other way around.

(In reply to WADA from comment #41)
> (a) Use other than Thunderbird as your mailer which fully fulfills your
> pretty important requirement(s).

Thunderbird is the closest, unfortunately.

> (b) Use other sophisticated and clever IMAP server who supports feature of
> SHOULD even if the feature is not MUST in RFC.

This seems to be the Mozilla way of dealing with bugs: Blame it on the others, no matter how. While my comment may not be constructive, neither is yours. We're living in a real world that doesn't always work as designed.

[M-wada-7 (m-wada-7)]

(In reply to Yves Goergen from comment #42)
> In case you're not aware of it anymore, going offline and back online does not resolve the issue. At least for me.
> The only thing that does help is restarting Thunderbird, which I have to doo every few days (snip)

Funny. It sounds that your Thunderbird is not standard one...
IIUC, IIRC, after Go Work Offline, Tb closes all connections, and after Go back Work Online, Tb executes login sooner or later, and after login, Tb opens Inbox(and others if requested) for new mail check by Biff, or open Inbox (or an Mbox) when selected, and upon first select Inbox/Mbox at a cached connection, Tb always issues "uid 1:* fetch Flags()" unless CONDSTORE is used. This is confirmed many times by getting IMAP log.
If "uid 1:* fetch Flags()" is used, Tb should know all flags of all mails in an Mbox.
If not, it usually means that your Thunderbird is not properly configured.

Have you checked IMAP protocol level flow by yourself?
Do you correctly disable CONDSTORE support as stated in this bug more than one time?
Or if Inbox, did you do required "click Inbox while other than Inbox, for example account, is selected at Folder Pane"?
Because "new mail check by Biff" is not immediately invoked after going back to Work Online"(invoked at next new mail check interval), "time lag" exists between "going back to Work Online" and "first new mail check after going back to Work Online".

> Thunderbird is the closest, unfortunately.
It's sad. We will have to see complants by you, which surely won't help problem resolving, many times in this bug....

[Vseerror (vseerror)]

*** Bug 526359 has been marked as a duplicate of this bug. ***

[Vseerror (vseerror)]

rsx11m, Nikolay, do you have anything to add?

[Nikolay Shopik (shopik)]

I don't use gmail+imap, but I've seen some issues with marking many years ago (these was never been consistent)

[Rsx11m-pub (rsx11m-pub)]

If I recall correctly, that's a case which the condstore feature is supposed to cover. However, since the MailNews implementation is broken (see bug 885220 and bug 912216) I wouldn't be surprised if it's an issue prompted by that. I don't see this for non-Gmail servers which are not using condstore, though changes in message status don't propagate immediately either.

[M-wada-7 (m-wada-7)]

*** Bug 1096022 has been marked as a duplicate of this bug. ***

[Raman Gupta (rocketraman)]

I've found that some extensions also cause problems with this. Two that I've found so far are:

- ImportExportTools (seems to do lots of concurrent IMAP operations with Tbird, often resulting in "Operation in progress" messages)
- Send Without Save(Copy Sent to Current has equivalent functionality and seems to be fine)

Once I disabled the above extensions, using a cyrus imapd 2.4 IMAP account that supports IDLE and CONDSTORE, Tbird changes from often requiring the Offline/Online trick, to usually updating immediately, sometimes requiring a folder change to see the updates, and occasionally requiring a Compact to see the updates. But as far as I have seen so far, Tbird now never requires Offline/Online to sync the folder, which is a huge improvement.

Disabling CONDSTORE further improves matters and Tbird more often (always?) updates the folder immediately. Obviously that is related to the separate CONDSTORE issues though.

[Alister Hood (alister-hood)]

(In reply to WADA from comment #43)
> (In reply to Yves Goergen from comment #42)
> > In case you're not aware of it anymore, going offline and back online does not resolve the issue. At least for me.
> > The only thing that does help is restarting Thunderbird, which I have to doo every few days (snip)
>
> Funny. It sounds that your Thunderbird is not standard one...

I don't know about Thunderbird, but I see the same thing (going offline does not work, I have to restart) in "standard" Seamonkey, and have done through many versions of Seamonkey. In Seamonkey it is actually more annoying, because you need to shutdown all Seamonkey windows (i.e. browser windows and everything, not just mail and news windows).

[M-wada-7 (m-wada-7)]

(In reply to alister.hood from comment #50)>
> but I see the same thing (going offline does not work, I have to restart) in "standard" Seamonkey, (snip)

IIUC, there is no difference between SeMonkey and Thunderbird, as for phenomeon of this bug.

(a) As repeatedly written, after go Work Offline the go back Work Online, explicit folder open by new folder click is mandtory to force server access again. When did go Work Offline the go back Work Online, did youa ctually do it?
      - If Inbox is already selected at folder pane when "go back Work Online" is executed,
          click account then click Inbox again at folder pane.
(b) As repearedly written, Gmail IMAP started to enable CONDSTORE and Tb's CONDSTORE support is enaabled by default.
      When CONDSTORe is used, Tb may not use "uid fetch 1:* flags" or "uid fetch 1:* flags CHANGEDSINCE nnnn".
      Tb may use "uid fetch <HighestUID>:* flags" or "uid fetch <HighestUID>:* flags CHANGEDSINCE nnnn".
      As repearedly written, to force "uid fetch 1:* flags" always in recovery operation for this bug,
      disbaling Tb's CONDSTORE support is needed for problem determination.
      Did you properly disabled Tb's CONDSTORE support?
      Read thru Bug 912216 and bug 885220, al repeatedly written.

Further, get IMAP log and check imap level flow by yourself, before adding comment like "annoying ..." which will never help activity for resolving problem.
Problem of this bug is pretty simple.
1. To know flag state change of already fetched mail of UID=NN<CurrentlyHighestUID, one of next is needed.
    1-1. flag staae change notification for the UID=NN from server via unsol response to lIDLE for selected Mbox.
            => Tb expects it, but maany IMAP servers including Gmail IMAP won't do this.
    1-2. uid fetch NN flags at connection where the Mbox is selected.
            => Tb uses "uid fetch CurrentlyKnownHighestUID:* flags" for new maail check of already selected Mbox.
                  So, flag state change of UID=NN can not be known untill "uid fetch 1:* flags" is issued again for the Mbox.
2. Recovery operation stated in this bug is simply "a procedure to try to force uid fetch 1:* flags".
    2-a. Unless you explicitly click Mbox, connection with server for the Mox won't be opened immediately.
            Without connection with imap server, any "uid fetch ..." won't be issued for the mbox by Tb.
    2-b. Even when connection for the Mbox is established and the Mbox is selected at connection,
             Tb may not issue uid fetch 1:* flags". Tb may use "uid fetch HighestUID:* flags ...".
             Even if "uid fetch 1:* ..." is used by Tb, it may be "uid fetch 1:* CHANGEDSINCE nnnn"
             where nnnn is too large(may be newer than MODSEQ for flag change of UID=NN).

[Alister Hood (alister-hood)]

(In reply to WADA from comment #51)
...
> (a) As repeatedly written, after go Work Offline the go back Work Online,
> explicit folder open by new folder click is mandtory to force server access
> again. When did go Work Offline the go back Work Online, did youa ctually
> do it?

Yes, did that.

> - If Inbox is already selected at folder pane when "go back Work
> Online" is executed,
> click account then click Inbox again at folder pane.

Did that too.

> (b) As repearedly written, Gmail IMAP started to enable CONDSTORE and Tb's
> CONDSTORE support is enaabled by default.
...
> Did you properly disabled Tb's CONDSTORE support?
> Read thru Bug 912216 and bug 885220, al repeatedly written.

Well, I disabled it in mail.server.default.use_condstore - there is no ...use_condstore setting for the account, so I guess Seamonkey doesn't implement it per account - or should I just create the setting?

> Further, get IMAP log and check imap level flow by yourself

Unfortunately that'll have to wait for another day.
But testing a few more times I discovered that it actually does work, just only every second time - i.e. I have to do "offline>online>other folder>back to folder>offline>online>other folder>back to folder" before it updates.

[Rsx11m-pub (rsx11m-pub)]

(In reply to alister.hood from comment #52)
> Well, I disabled it in mail.server.default.use_condstore - there is no
> ...use_condstore setting for the account, so I guess Seamonkey doesn't
> implement it per account - or should I just create the setting?

The implementation is identical between SeaMonkey and Thunderbird. If there is no server#-specific setting mail.server.id#.use_condstore then mail.server.default.use_condstore is used. Thus, setting mail.server.default.use_condstore to false should disable it for all configured IMAP servers unless specific preference settings for that server override it.

[M-wada-7 (m-wada-7)]

(In reply to alister.hood from comment #52)
> But testing a few more times I discovered that it actually does work, just only every second time
> - i.e. I have to do "offline>online>other folder>back to folder>offline>online>other folder>back to folder" before it updates.

It's perhaps "connection loss during IDLE".
  IDLE -> Sm/Tb(and server) enters "receive state" -> Work Offline -> Tb closes server connection(Socket, TCP/IP level)
  Tb perhaps closes IMAP connection. But for SeaMonkey, this is perhaps "connection loss during IDLE".
  By click of Inbox again, Sm tries to send DONE(enters Send mode)
  -> because TCP session is already closed by Sm, nothing is sent to server, and Sm waits for timeout.
  -> go Work Offline/Online again -> send of DONE fails and imap connection is normally closed in Sm
  -> by next click of Inbox, TCP session is established and imap connection is established
It seems that error detection code is different between SeaMonkey and Thunderbird.

> so I guess Seamonkey doesn't implement it per account - or should I just create the setting?

There is no need of guessing. "CONDSTORE is aactually used or not" is always clearly known pretty easily by simply looking imap log.

[M-wada-7 (m-wada-7)]

FYI #1.
If CONDSTORE support of Tb is enabled(mail.server.default.use_condstore=true, or mail.server.server#.use_condstore=true), and if imap server supports CONDSTORE, bug 1123617 occurs even after go Work Offline/go Work Online again.
i.e. "Workaroudof this bug/recovery operaation of this bug", go Work Offline/go Work Online again/explicitly open Mbox again, won't work due to bug 1123617 .
FYI #2.
If bug 1123617 will be fixed, this bug will be resolved by utilizing CONDSTORE, because change required for that bug is "uid fetch 1:* Flags (CHANGEDSINCE LargestKnowMODSEQ) upon each new mail check".

[M-wada-7 (m-wada-7)]

FYI.
I've opened Bug 1124924.
Bug 1124924 - If CONDSTORE is usable, use "uid fetch 1:* flags (CHANGDEDSINCE modseq)" for new mail check by Biff instead of "uid fetch NextUID:* flags", in order to resolve problem like Bug 693204, Bug 517461

[Guyette (guyette)]

All this discussion is very well and good, but how do we work around this problem?

I'm deleting e-mails from Gmail on my iPhone and still seeing them hours later in Thunderbird.

Work Offline > Work Online > Click another folder > Click Inbox seems to force resynch.

I'd really prefer not to do this by hand every hour.

What is the prescribed fix / workaround?

[M-wada-7 (m-wada-7)]

(In reply to Tom Guyette from comment #57)

Do you read about a workaround of "max cached connection=1"?

If shared with iPhone, I believe mail volume, number of mboxes, frequency of new mails, new mail volue, are not so large,
If so, "max cached connection=1", "disable IDLE", "adequate new mail check intervlal", can be used.
Needless to say, "disabling CONSTORE" is mandatory.
1. Click other than Inbox(call FolderX) -> SELECT FolderX -> Inbox is unselected at only one connection
2. Soner or later, new mail check occurs -> SELECT Inbox -> Inbox is selected at only one connection -> uid fetch 1:* Flags
3. When you need to snch flsag state with Gmail IMAP and iPhone, click FolderX, then click Inbox
     SELECT FolderX -> SELECT Inbox -> uid fetch 1:* Flags for Inbox is issued.
     As I repeadly staated, workaround explained in this bug is to force this "uid fetch 1:* Flags" for Inbox.

Another way: Use Inbox as mail drop, mail tray. Don't use Inbox as mail repository for log time.
                       Becuase of IMAP many Mboxes are lready supported. There is no need of folder named "Inbox" for mail viewing.

Because Inbox is mbox where new mails arrives, Tb always keeps Inbox "selected state" at a cached connection for Inbox only, and kepp Inbox open.
This is a reason why "uid fetch 1:* Flags" is not issued for Inbox while Tb is running.
If other folder than Inbox and Trash, connection for a FolderX is stolen by other FolderY when other FolderY is opened.
So, "max cached connections = 3 with folder switch of some folders someties" usually forces "uid fetch 1:* Flags" of FolderX, FolderY.

Why no problem in iPhone :
   Because of small/mobile device, connection is not kept for long time, and number of connections is usually one..
   Upon mail reading, login, SELECT Inbox, "uid fetch 1:* Flags" is executed.

[LonelyPixel (nospam-unclassified)]

Not sure what smartphones you have there but I have a permanent IDLE connection from mine, as on the desktop with Thunderbird. If disabling IDLE in Thunderbird is part of the workaround, then I think IDLE implementation is the problem. I'm not using IDLE for fun, I want to see new messages as they arrive. That's how modern event-driven systems work. Polling is from the 90s.

But since nobody was able to even locate the problem as of today, and given the open secret that Thunderbird is merely in maintenance mode for quite some time, I don't believe anybody is able to fix this issue (and others) without rewriting it all. If only I had more time...

[M-wada-7 (m-wada-7)]

(In reply to Yves Goergen from comment #59)
> Not sure what smartphones you have there but I have a permanent IDLE connection from mine,
> as on the desktop with Thunderbird. If disabling IDLE in Thunderbird is part of the workaround, then I think IDLE implementation is the problem.
> I'm not using IDLE for fun, I want to see new messages as they arrive. That's how modern event-driven systems work. Polling is from the 90s

Do you understand workaround of "max cached connection=1" and understaand IDLE?
If "max cached connection"=1, and if "other than Inbox, call FolderX" is selected at the only one connection, IDLE is effective only for FolderX. If Inbox is de-selectied at yhe only one cached connection, IDLE can do nothing on Inbox. If workround of "max cached connection"=1 is used, "IDLE works on Inbox sometime but IDLE won't work on Injbox other times" is pretty confusing, and it may produce useless bug open at B.M.O.

> I don't believe anybody is able to fix this issue (and others) without rewriting it all. If only I had more time.

You are absolutely free to believe anything.

Have you read and understood all bugs pointed in Bug 912216?
Because Gmail IMAP started to support CONDSTOREl last year, "solution of this bug utilizing CONDSTORE" is possible. As known by bugs pointed in Bug 912216, code change needed for the solution is not so huge. Rather, relatively small change than we thought.
Neeedless to say, as known by Bug 912216, there is problem in current CONDSTORE support in Tb, so it should be resolved.
And, for better solution, QRESYNC support by both Gmal IMAP and Tb is needed.

[Guyette (guyette)]

(In reply to WADA from comment #58)
> (In reply to Tom Guyette from comment #57)
> Do you read about a workaround of "max cached connection=1"?

Thanks WADA. I read the whole thread, after reading another long thread that was closed as a duplicate of this one, so by the time I got here I was very tired and just wanted deleted messages to go away.

Max cached connections was the only part in all that reading that I understood, so I have max connections set to 1. It slows down my client a bit, but I can accept that.

> If so, "max cached connection=1", "disable IDLE", "adequate new mail check intervlal", can be used.

I didn't understand what "disable IDLE" was until I started writing the proposed steps below. I still don't understand what "adequate new mail check interval" means. Bear in mind, I don't understand the client as a developer, I understand it as a casual user, so I need click-by-click, press-by-press, key-by-key instructions. I suspect most other users who would come across this thread would need that too, and ideally it would be pinned to the top of the bug. I've made an attempt at documenting the prescribed workaround below.

> Needless to say, "disabling CONSTORE" is mandatory.

I don't understand what that means, so it's not included in the steps below.

> 1. Click other than Inbox(call FolderX) -> SELECT FolderX -> Inbox is
> unselected at only one connection
> 2. Soner or later, new mail check occurs [...]

I did an experiment and I believe "sooner or later" can be replaced with "Restart Thunderbird." This is my attempt at documenting the workaround step by step:

FIRST, RECONFIGURE YOUR ACCOUNT:
1 - From the main Thunderbird screen, select "Tools" menu > "Account Settings" to launch the Account Settings window.
2 - On the left side of the Account Settings window, select "Server Settings" under your Gmail account name to view the Server Settings page.
3 - On the Server Settings page, press the "Advanced..." button to launch the Advanced Server Settings popup window.
4 - Set "Maximum number of server connections to cache" to 1.
5 - Deselect the checkbox for "Use IDLE command if the server supports it."
6 - Press the OK button to close the Advanced Server Settings window.
7 - Press the OK button to close the Account Settings window.

EVERY TIME YOU WANT TO REFRESH THE INBOX:
1 - Select the "Sent Mail" folder (or any other subfolder under the affected account).
2 - Close Thunderbird, including all mail viewing windows.
3 - Start Thunderbird.
4 - Select the "Inbox" folder of the affected account. Thunderbird should completely refresh the Inbox.

> [...] -> SELECT Inbox -> Inbox is
> selected at only one connection -> uid fetch 1:* Flags
> 3. When you need to snch flsag state with Gmail IMAP and iPhone, click
> FolderX, then click Inbox
> SELECT FolderX -> SELECT Inbox -> uid fetch 1:* Flags for Inbox is
> issued.
> As I repeadly staated, workaround explained in this bug is to force
> this "uid fetch 1:* Flags" for Inbox.
>
> Another way: Use Inbox as mail drop, mail tray. Don't use Inbox as mail
> repository for log time.
> Becuase of IMAP many Mboxes are lready supported.
> There is n...

[M-wada-7 (m-wada-7)]

(In reply to Tom Guyette from comment #61)
> Would it be possible to simply add a "Refresh folder" function to the "Get Messages" dropdown button
> that would perform a complete refresh of the current folder and Preferred Folder?

Tb already has button for "Refresh folder from scratch". It's called "Repair Folder" button :-)

> Better yet, add a Refresh button in the same row as Unread, Starred, Contact, Tags, and Attachment?

You are right.
There is no way to intentionally close mail folder in Tb.There is no way to force de-select of Mbox at a connection.
There is no way to force "uid fetch 1:* Flags" to recover from this bug by simple/easy operation.
Purpose of any workaround stated in this bug is : force "uid fetch 1:* Flags" for Inbox.
    Go Offline/Go Online : Force logoff -=> Tb has to do login, SELECT Inbox, uid fetch 1:* Flags
    max cached connections=1 : FolderX steels connection from Inbox => When Inbox is accessed, Tb has to do SELECT Inbox, uid fetch 1:* Flags
If "close/reopen folder" like button/menu will be implemented, it's helpful for this bug.
But there is no need of such button in usual use. Button only for working around problem(in this case, in both Tb and Gmail) is not usually implemented.

About IDLE setting : Server Settings/Advanced
About CONDSTORE : read Bug 912216, please.
About "adequate new mail check interval" :
   If "a few new mails per an hour", I believe "Check new messages Every 1 minute" is usually not mandatory.
   If "10000 new mails per an hour", I believe "Check new messages Every 1 minute" is not needed(rather annoying),
   because you can know about many many new mails upon each mail viewing. :-)

[LonelyPixel (nospam-unclassified)]

(In reply to Tom Guyette from comment #61)
> I did an experiment and I believe "sooner or later" can be replaced with
> "Restart Thunderbird." This is my attempt at documenting the workaround
> step by step:

> EVERY TIME YOU WANT TO REFRESH THE INBOX:

> 2 - Close Thunderbird, including all mail viewing windows.
> 3 - Start Thunderbird.

If I understood your headings, I can safely reduce all of your steps to the ones I left in my quotation. I see deleted mails almost every day in my primary mailbox which is not Gmail. I just need to restart Thunderbird and everything works nicely again. No other configuration is required for this to work. Still it's cumbersome to restart applications to repair them, it might as well crash every day. This really should not be necessary at all.

[Aaron-hastings11 (aaron-hastings11)]

Still seems to be an issue for me with Gmail. At least I assume this is the same issue. In my case, I filter all emails into folders (or "labels" as Gmail calls them) on the server. So all emails sent directly to *my* email address go to a folder/label called "Me".

== Steps to reproduce ==
1. Open Thunderbird.
2. Open arbitrary email app on phone/other computer. Send email to Gmail account.

== Expected results ==
Within seconds, Thunderbird displays the new mail in the "Me" folder, complete with a notification.

== Actual results ==
<nothing happens>

== Completely manual "workaround" ==
Click the "Me" folder. Thunderbird instantly displays the new message.

I note that even clicking "Get Messages" doesn't pull down the new message. What the heck? Clicking the folder gets a new message from the server but clicking "Get Messages" doesn't?

I have tried what I think is everything from this thread: disabled IDLE and changed max_cached_connections to 1. My CONDSTORE was actually "false" by default (Thunderbird 38.2.0 on OS X) so I left that as-is. I did a complete Exit of Thunderbird and restarted. No difference whatsoever.

[Vseerror (vseerror)]

*** Bug 1030044 has been marked as a duplicate of this bug. ***

[Vseerror (vseerror)]

*** Bug 987626 has been marked as a duplicate of this bug. ***

[Philippe Cloutier (chealer-gmail)]

I have had 2 Thunderbird 45 instances for the same Gmail account open in the last weeks, viewing the All Mail folder while I was deleting mails from that folder. On several occasions, I noticed after deleting and/or tagging mails from one instance that the other instance wasn't updated. The deleted mails would still show, and would keep showing even after "Get Messages". There was no activity shown, and even Activity Manager showed nothing going on. If I remember correctly, this fixed between 1 and 5 minutes after going back to the outdated instance (Thunderbird would take the initiative to update, even though it had been asked before).

It seems like after a while Thunderbird goes in some kind of sleep mode, and has trouble waking up from that, but it hasn't happened often enough for me to explain the precise conditions needed for this to happen.

I do not fully understand Summary and am unable to parse several comments, and as it would take an eternity to go through all other comments, I will assume that the symptoms I experienced are results of the bug reported here and add a vote.

[Vseerror (vseerror)]

*** Bug 945493 has been marked as a duplicate of this bug. ***