vm-imap-retrieved-messages for IMAP spool

[Uday Reddy]

The variable vm-imap-retrieved-messages stores a list of all the messages retrieved from the IMAP server. This list grows and grows endlessly. Stuffing this data into the folder header becomes time-consuming.

The list should be trimmed before stuffing into the folder, based on the messages currently known to be on the IMAP server.

But it is not clear if this is good enough for folders that use IMAP servers as spools.

[Uday Reddy (reddyuday)]

Info from "mere user"

I noticed that saving my inbox folder takes a very long time (9000+
messages). I get a message "stuffing folder data" and then a long
wait. This message is printed by vm-write-file-hook, so I tried to
profile the functions that are called by this function. Here are the
results. I wonder if this makes sense to you and if there is anything
that can be done to further diagnose and perhaps even optimize this?
Best, Eli

vm-stuff-imap-retrieved 1 14.723393 14.723393
vm-stuff-bookmark 1 0.060505 0.060505
vm-stuff-header-variables 1 0.026999 0.026999
vm-stuff-labels 1 0.026387 0.026387
vm-stuff-summary 1 0.026363 0.026363
vm-stuff-pop-retrieved 1 0.025692 0.025692
vm-update-summary-and-mode-line 13 0.008983 0.000691
vm-stuff-last-modified 1 0.005488 0.005488
vm-write-file-hook 1 3e-06 3e-06

[mere user (emacs-user)]

Hi Uday, I didn't realize you had such a precise idea of what the
issue is. that's quite exciting. is there anything I can do? what
does "good enough" below mean? in my case messages are simply
downloaded to my local computer and nothing is left on the imap
server. thanks for looking into this! E

On Sat, Dec 11, 2010 at 2:41 PM, Launchpad Bug Tracker
<email address hidden> wrote:
> You have been subscribed to a public bug by Uday Reddy (reddyuday):
>
> The variable vm-imap-retrieved-messages stores a list of all the
> messages retrieved from the IMAP server.  This list grows and grows
> endlessly.  Stuffing this data into the folder header becomes time-
> consuming.
>
> The list should be trimmed before stuffing into the folder, based on the
> messages currently known to be on the IMAP server.
>
> But it is not clear if this is good enough for folders that use IMAP
> servers as spools.
>
> ** Affects: vm
>     Importance: Low
>     Assignee: Uday Reddy (reddyuday)
>         Status: Triaged
>
> --
> vm-imap-retrieved-messages
> https://bugs.launchpad.net/bugs/671940
> You received this bug notification because you are a direct subscriber of the bug.
>

[mere user (emacs-user)]

so given your insight, I manually deleted most of

X-VM-IMAP-Retrieved:

in my inbox. the improvement is nothing but astonishing. thanks!! Eli

On Sat, Dec 11, 2010 at 2:41 PM, Launchpad Bug Tracker
<email address hidden> wrote:
> You have been subscribed to a public bug by Uday Reddy (reddyuday):
>
> The variable vm-imap-retrieved-messages stores a list of all the
> messages retrieved from the IMAP server.  This list grows and grows
> endlessly.  Stuffing this data into the folder header becomes time-
> consuming.
>
> The list should be trimmed before stuffing into the folder, based on the
> messages currently known to be on the IMAP server.
>
> But it is not clear if this is good enough for folders that use IMAP
> servers as spools.
>
> ** Affects: vm
>     Importance: Low
>     Assignee: Uday Reddy (reddyuday)
>         Status: Triaged
>
> --
> vm-imap-retrieved-messages
> https://bugs.launchpad.net/bugs/671940
> You received this bug notification because you are a direct subscriber of the bug.
>

[Uday Reddy (reddyuday)]

mere user writes:

> Hi Uday, I didn't realize you had such a precise idea of what the
> issue is. that's quite exciting. is there anything I can do? what
> does "good enough" below mean? in my case messages are simply
> downloaded to my local computer and nothing is left on the imap
> server. thanks for looking into this! E

I don't know, really. The code for IMAP spool files is what we
inherited from Kyle. I didn't really touch it. I only worked on IMAP
server folders code. In the process, I left out the handling of
vm-imap-retrieved list because it was being shared by both pieces of
code, and I didn't want break anything.

Now that you say it is not working for the IMAP spool files either, I
think I should get back to it and fix it. If you are leaving no email
on the server, it is perfectly fine to trim the list manually so that
it speeds up. I am glad you figure it out!

Cheers,
Uday

[Uday Reddy (reddyuday)]

The X-VM-IMAP-Retrieved header is not being used in the server folders code. It is only used in IMAP spool files.

A bug in vm-imap-clear-invalid-retrieval-entries was fixed in revno. 990. This should help clear away the old retrieved-messages entries when the UIDVALIDITY changes on the server. But the expunge operations are yet to be fixed.

[Uday Reddy (reddyuday)]

Oops, that should be revision 999.

[mere user (emacs-user)]

thanks... as far as I can tell the X-VM-IMAP-Retrieved header still
grows with new retrieved messages.

On Sat, Dec 25, 2010 at 8:32 PM, Uday Reddy <email address hidden> wrote:
> The X-VM-IMAP-Retrieved header is not being used in the server folders
> code.  It is only used in IMAP spool files.
>
> A bug in vm-imap-clear-invalid-retrieval-entries was fixed in revno.
> 990.  This should help clear away the  old retrieved-messages entries
> when the UIDVALIDITY changes on the server.  But the expunge operations
> are yet to be fixed.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/671940
>
> Title:
>  vm-imap-retrieved-messages
>
> Status in VM (View Mail) for Emacs:
>  Triaged
>
> Bug description:
>  The variable vm-imap-retrieved-messages stores a list of all the messages retrieved from the IMAP server.  This list grows and grows endlessly.  Stuffing this data into the folder header becomes time-consuming.
>
> The list should be trimmed before stuffing into the folder, based on the messages currently known to be on the IMAP server.
>
> But it is not clear if this is good enough for folders that use IMAP servers as spools.
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/vm/+bug/671940/+subscribe
>

[Uday Reddy (reddyuday)]

Can you turn off vm-imap-expunge-after-retrieving and issue
vm-expunge-imap-messages manually? See if that works right?

Cheers,
Uday

[mere user (emacs-user)]

set to nil and expunged manually as suggested below. it didn't
eliminate the X-VM-IMAP-Retrieved from the inbox file, and new entries
are still added to it. E

On Sun, Dec 26, 2010 at 11:44 AM, Uday Reddy <email address hidden> wrote:
> Can you turn off vm-imap-expunge-after-retrieving and issue
> vm-expunge-imap-messages manually?  See if that works right?
>
> Cheers,
> Uday
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/671940
>
> Title:
>  vm-imap-retrieved-messages
>
> Status in VM (View Mail) for Emacs:
>  Triaged
>
> Bug description:
>  The variable vm-imap-retrieved-messages stores a list of all the messages retrieved from the IMAP server.  This list grows and grows endlessly.  Stuffing this data into the folder header becomes time-consuming.
>
> The list should be trimmed before stuffing into the folder, based on the messages currently known to be on the IMAP server.
>
> But it is not clear if this is good enough for folders that use IMAP servers as spools.
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/vm/+bug/671940/+subscribe
>

[Uday Reddy (reddyuday)]

Fixes made in revision 1000.

[mere user (emacs-user)]

works fine for me. no more entries are added. if entries exist
already, they are not eliminated. thanks as always! E

On Tue, Dec 28, 2010 at 7:29 PM, Uday Reddy <email address hidden> wrote:
> Fixes made in revision 1000.
>
>
> ** Changed in: vm
>       Status: Triaged => Fix Committed
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/671940
>
> Title:
>  vm-imap-retrieved-messages
>
> Status in VM (View Mail) for Emacs:
>  Fix Committed
>
> Bug description:
>  The variable vm-imap-retrieved-messages stores a list of all the messages retrieved from the IMAP server.  This list grows and grows endlessly.  Stuffing this data into the folder header becomes time-consuming.
>
> The list should be trimmed before stuffing into the folder, based on the messages currently known to be on the IMAP server.
>
> But it is not clear if this is good enough for folders that use IMAP servers as spools.
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/vm/+bug/671940/+subscribe
>

[Uday Reddy (reddyuday)]

mere user writes:

> works fine for me. no more entries are added. if entries exist
> already, they are not eliminated. thanks as always! E

You are welcome. Apparently it was a bug that I had introduced
myself. The original Kyle's system didn't have the problem. So I am
glad I tracked it down now.

The existing entries can't be eliminated easily because VM has to be
sure that they have been expunged on the server. But they will go
away eventually when the UIDVALIDITY on the server changes.

Cheers,
Uday

[Uday Reddy (reddyuday)]

Dear Mark, I am posting here a patch to vm-imap.el which adds a new command called `vm-prune-imap-retrieved-list'.

I would appreciate if you can test it for me by periodically running it to get rid of the extra entries in the imap-retrieved-list. (This is assuming that you will continue to use VM 8.1.93a. If you use the trunk version, there will be nothing for this command to do basically.)

[Uday Reddy (reddyuday)]

Here is an improved patch.

The third hunk fails when I apply to 8.1.93a, but you can ignore that.

[Mark Diekhans (markd-kermodei)]

The patch references a missing function:
    Symbol's function definition is void: vm-mark-folder-modified-p

Note, I am happy to switch to the development tree or continue
to test this patch. Which ever is more helpful to the project.

Mark

[Uday Reddy (reddyuday)]

You are right, Mark. I was a bit too optimistic about how easily this
patch might fit into 8.1.x. I will have to do a proper backport. I
will keep you posted.

Cheers,
Uday

[Uday Reddy (reddyuday)]

Hi Mark, here is yet another patch, tested this time. Thanks for your help!

Uday

[Mark Diekhans (markd-kermodei)]

New patch works great.

Many thanks!!!