Error visiting large mail files wiht threading turned on

[Tim Cross]

When visiting some mail folders I get this error

Debugger entered: (spurious [[#<marker at 17203581 in mlinnema> #<marker at 17203614 in mlinnema> nil #<marker at 17205892 in mlinnema> #<marker at 17209467 in mlinnema> #<marker at 17209468 in mlinnema>] ["500" "500" nil nil nil <<>> <-- From_ "3222" #<buffer mlinnema> nil (<013f01c7a341$0ac2f8a0$2048e9e0$@edu\.au> <18014\.24020\.714365\.408130@blind-bat\.une\.edu\.au> <014a01c7a346$8970fcb0$9c52f610$@edu\.au>) nil nil nil nil nil nil nil nil nil] [nil nil nil nil t nil nil nil nil nil nil nil nil nil nil nil] ["3575" "Thursday" "31" "May" "2007" "15:42:55" "+1000" "Martina Linnemann" "<email address hidden>" "<014a01c7a346$8970fcb0$9c52f610$@edu.au>" "92" "RE: \"Welcome\" email to new users" "^From:" "<email address hidden>" "'Tim Cross'" "5" "2007053115:42:55" "\"Welcome\" email to new users" (number " " mark " R Martina Linnemann May 31 92/3575 " thread-indent "\"RE: \"Welcome\" email to new users\"
") "<email address hidden>" ("<013f01c7a341$0ac2f8a0$2048e9e0$@edu.au>" "<email address hidden>") nil nil nil nil nil] [nil <v> nil nil nil nil]])
  #[(m) "\305!\203

If I start vm without threading enabled, I can visit the folder. If I've started VM with threading enabled, visiting gives the above backtrace . Visiting a second time seems to work. If I do C-t to turn threading off, it still attempts to visit the new folder with threading turned on (should C-t turn threading off globally or just per folder?). The folder in question is a bit over 22MB and I've seen the same error on other largish folders (I actuallly don't consider 22MB a particularly large folder).

This is with most recent VM revno 1138 under emacs 24.0.50

[Uday Reddy (reddyuday)]

Hi Tim, note that your backtrace has gotten cut off.

The problem is that the threads database is getting corrupted or bcoming faulty somehow. This is an integrity check running and throwing up an alarm.

For the time being, you can type `c' (for continue) when this happens, and it will try to rebuild the threads database. There is no guarantee that the rebuilt database will not be faulty again. If the problem happened because of some previous operations then the rebuilt database will be fine.

But, if it is some pattern in the folder that is causing the fault then it might happen persistently. The only way out at the moment is to revisit the folder afresh without threading.

[Tim Cross (tcross)]

On Wed, Mar 9, 2011 at 9:09 PM, Uday Reddy <email address hidden> wrote:

> Hi Tim, note that your backtrace has gotten cut off.
>
> The problem is that the threads database is getting corrupted or bcoming
> faulty somehow. This is an integrity check running and throwing up an
> alarm.
>
> For the time being, you can type `c' (for continue) when this happens,
> and it will try to rebuild the threads database. There is no guarantee
> that the rebuilt database will not be faulty again. If the problem
> happened because of some previous operations then the rebuilt database
> will be fine.
>
> But, if it is some pattern in the folder that is causing the fault then
> it might happen persistently. The only way out at the moment is to
> revisit the folder afresh without threading.
>
>
OK, I'll try to debug it further when i get time.

Some additional things to note

1. This occurs on larger folders which have never been visited before. I
guess this would indicate it is a problem with the initial build of the
threads database (assuming each folder thread db is independent).

2. It seems to happen with all slightly larger folders, so while it still
may be some message which is 'unusual' or has something that the code cannot
handle, this seems less likely.

3. In all the cases I've run into this, no other operations have occured in
that folder i.e. not visited it, done something and then visited it again. I
don't seem to see this problem with smaller folders, but don't know yet if
there is a 'cut-off' size where this begins to occur.

Tim

[Uday Reddy (reddyuday)]

Tim, I have made the entry into the debugger conditional on the flag `vm-thread-debug'.

If you want to continue debugging it, please turn this flag on in your init file.

Cheers,
Uday

[Uday Reddy (reddyuday)]

Hi Tim, I fixed a nasty bug with threading, but it would only happen when there were multiple virtual folders on the same real folder. So, this won't probably apply to your case.

Do you have any more information about what is provoking the bug for you? Note that you haven't yet sent me a good backtrace. You can only get the backtrace if set vm-thread-debug to t. If not, it just recovers from the errors as best as it can.

To turn off threading when you visit folders you need to use
  (setq-default vm-summary-show-threads nil)
C-t only works inside a single folder.

Cheers,
Uday

[Tim Cross (tcross)]

On Fri, Mar 18, 2011 at 11:04 AM, Uday Reddy <email address hidden>wrote:

> Hi Tim, I fixed a nasty bug with threading, but it would only happen
> when there were multiple virtual folders on the same real folder. So,
> this won't probably apply to your case.
>
> Do you have any more information about what is provoking the bug for
> you? Note that you haven't yet sent me a good backtrace. You can only
> get the backtrace if set vm-thread-debug to t. If not, it just recovers
> from the errors as best as it can.
>
> To turn off threading when you visit folders you need to use
> (setq-default vm-summary-show-threads nil)
> C-t only works inside a single folder.
>
>
Hi Uday, no, I have not had time to look at this again. Will try to send a
full backtrace sometime over the weekend.

Tim

[Uday Reddy (reddyuday)]

I have done some work on threads now, and I believe I may have fixed the common integrity check failures. Revision 1174.

Kyle Jones allowed cycles in thread trees (which come from bad headers in email messages), but broke the cycles one way or the other when he traversed the trees to calculate thread ancestry chains. I think this gave rise to inconsistent threading information because different orders of building the ancestry chains might break the cycles in different ways.

Following Jamie Zawinski's recommendation, I am now avoiding cycles in thread trees. So, as long as a particular threads database is active, it would have the cycles resolved in exactly the same way. Thanks to the GNU bug tracker which provided me an example of bad headers, allowing me to investigate and debug this problem.

So, please try it out on all your bad folders, setting vm-thread-debug to t. If the thread integrity check fails, you will get a debug entry point, and you would need to send me the backtrace. (Better to send it to me by email. Cutting and pasting backtraces into the Launchpad web forms doesn't work right.)

[Uday Reddy (reddyuday)]

I am pusing some more bug fixes for threading in revision 1180. If it causes any problems, please back up to revision 1174, which seems to be stable.

  bzr revert -r1174

[mere user (emacs-user)]

thanks for the warning! seems to work very well with a folder of
12000 messages and with threading on or off. Eli

On Tue, Apr 5, 2011 at 8:05 AM, Uday Reddy <email address hidden> wrote:
> I am pusing some more bug fixes for threading in revision 1180.  If it
> causes any problems, please back up to revision 1174, which seems to be
> stable.
>
>  bzr revert -r1174

and especially thanks for this reminder...

[Uday Reddy (reddyuday)]

mere user writes:

> thanks for the warning! seems to work very well with a folder of
> 12000 messages and with threading on or off. Eli

The problems are not just from size, but from messages with bad
headers, especially the "In-Reply-To" and "References" headers. There
could also be some strange cases that have good headers but VM might
not deal with them correctly. The bigger the folders, the more likely
it is that they have the strange cases.

In the past, it didn't matter much if the threading inside VM went
wrong. But I have been sprucing up the threading code lately because
we are doing thread operations for which correct threads are
essential. Thus all the strange problems are showing up.

Cheers,
Uday

[mere user (emacs-user)]

I did notice that unrelated messages are treated as a single thread:
say I have several independent message months apart, that just happen
to have a common subject ("lunch"), but are not part of the same
conversation and not even with the same correspondents, then they seem
to be treated as a single thread. perhaps that's intended, though. E

On Wed, Apr 6, 2011 at 7:47 AM, Uday Reddy <email address hidden> wrote:
>
> The problems are not just from size, but from messages with bad
> headers, especially the "In-Reply-To" and "References" headers.  There
> could also be some strange cases that have good headers but VM might
> not deal with them correctly.  The bigger the folders, the more likely
> it is that they have the strange cases.
>
> In the past, it didn't matter much if the threading inside VM went
> wrong.  But I have been sprucing up the threading code lately because
> we are doing thread operations for which correct threads are
> essential.  Thus all the strange problems are showing up.
>
> Cheers,
> Uday

[Arik (akwm)]

This sounds like you have vm-thread-using-subject set to t (the
default), I haven't found this to be necessary these days so I'm not
sure why the default is t...

-Arik

[Uday Reddy (reddyuday)]

mere user writes:

> I did notice that unrelated messages are treated as a single thread:
> say I have several independent message months apart, that just happen
> to have a common subject ("lunch"), but are not part of the same
> conversation and not even with the same correspondents, then they seem
> to be treated as a single thread. perhaps that's intended, though. E

Hmm, if you are only noticing it now, that means that the threading is
working a lot more effectively now than it did in the past.

You could set vm-thread-using-subject to nil, as Arik mentioned. Or,
you can have more fun by using thread folding, which will hide all the
old junk, while still telling you that it is there. To enable it, set
vm-summary-enable-thread-folding to t.

Cheers,
Uday

[Uday Reddy (reddyuday)]

John Hein reports that 1174 has an arithmetic error, which is true. So, please 1175 as the fall back option if need be.

But the current revision, 1181, seems be working ok.

[Uday Reddy (reddyuday)]

I made another push for threading code, in revision 1184. If it proves to be problematic, please back up to revision 1183.

[Uday Reddy (reddyuday)]

I am closing this bug report for now. The testing and debugging of threading stuff will continue.