Crash in groupcompress during commit operation

[Gareth White]

While running "bzr commit" bzr.exe crashed (hard) with an access violation. I was able to save a crash dump and load it in Windbg, which indicated it was crashing in bzrlib__groupcompress_pyx!init_groupcompress_pyx.

I ran the same command again and it crashed in the same way. I then turn on some of the debug flags and ran it again, but this time it didn't crash! I'll attach the logs I've got anyway.

Is there anything I should do to make it easier to debug the crash if it occurs again?

This crash was with Bazaar 2.0.1 running on Windows XP SP3.

---

the stack trace claims to be

STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be wrong.
0012e4a8 01b02035 020f0080 03b7ebd0 01b08188 bzrlib__groupcompress_pyx!init_groupcompress_pyx+0x12b9
0012e4f8 1e08f24b 03951a20 0378c918 00000000 bzrlib__groupcompress_pyx+0x2035
0012e514 1e03b442 0530b260 0378c918 00000000 python25!PyCFunction_Call+0x5b
0012e53c 1e03cf85 1e03d4a3 0012e56c 00000000 python25!PyEval_CallObject+0x2b2
0012e540 1e03d4a3 0012e56c 00000000 015284e8 python25!PyEval_EvalFrameEx+0x18c5
0012e58c 1e0d99db 00a434b0 1e071215 ffffffff python25!PyEval_EvalFrameEx+0x1de3
0012e594 1e071215 ffffffff ffffffff 00000000 python25!PyType_Lookup+0x3feb
0012e5ac 1e03e3ed 07c92e78 00000000 01513420 python25!PyFrame_New+0x75
0012e5e0 1e03a29c 015284e8 07c92e78 00000000 python25!PyEval_EvalCodeEx+0x62d
00332da0 00332d58 07c92e78 0000001a 00000000 python25!PyEval_GetFuncDesc+0x17c
00000000 00000000 00000000 00000000 00000000 0x332d58

[John A Meinel (jameinel)]

I'm guessing the stack trace is wrong.
0012e4a8 01b02035 020f0080 03b7ebd0 01b08188 bzrlib__groupcompress_pyx!init_groupcompress_pyx+0x12b9

"init_groupcompress_pyx" is the module load function.
It has gotten to the point of "Auto-packing ..." before it failed, which is long after we've imported the module.

Looking at the rest, it seems to be claiming that "instruction at 0x00 tried to read from 0x03b7f000 and failed". Though it would make a lot more sense if that was reversed, and we tried to read from a NULL pointer.

My quick guess is that this is an Out-of-memory failure that we didn't handle properly (malloc returns NULL but we don't check before returning, etc.)

I did some looking around, but all the places appear to do the right thing and check the return value of malloc. So the next thing to check is that if the callers all check their returns, but looking around everything seems ok.

You might be able to force the failure by running "bzr pack". You may not provoke the specific failure, but if you get an out-of-memory exception, then it is fairly likely that is the original failure.

[Robert Collins (lifeless)]

FWIW we're seeing segfaults on a PQM instance, but haven't isolated the
cause to python || bzr's modules yet. This may be the same issue.

[Gareth White (gwhite-deactivatedaccount)]

I got the same crash again last night so it's not just a once-off issue. However, it is difficult to reproduce. I'm importing a lot of data into a large repository and so far it's crashed on revision 5709 and 8409.

I tried "bzr pack" today but it didn't cause it to crash again. The peak memory usage during the pack was 479 MB, so it's unlikely to have really run out of memory.

I'm going to try it on a different machine and leave it running overnight again.

Is there a way to get a better stack trace from the crash? Would debugging be easier if I built Bazaar from scratch instead of using the redistributable?

[Martin Pool (mbp)]

2009/12/1 Gareth White <email address hidden>:
> I got the same crash again last night so it's not just a once-off issue.
> However, it is difficult to reproduce. I'm importing a lot of data into
> a large repository and so far it's crashed on revision 5709 and 8409.
>
> I tried "bzr pack" today but it didn't cause it to crash again. The peak
> memory usage during the pack was 479 MB, so it's unlikely to have really
> run out of memory.
>
> I'm going to try it on a different machine and leave it running
> overnight again.
>
> Is there a way to get a better stack trace from the crash? Would
> debugging be easier if I built Bazaar from scratch instead of using the
> redistributable?

It could be. John, is there any way for him to build it with debug
symbols included? Or indeed could we perhaps ship them in our
standard build? It's been years and years since I dealt with msvc.

--
Martin <http://launchpad.net/~mbp/>

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Pool wrote:
> 2009/12/1 Gareth White <email address hidden>:
>> I got the same crash again last night so it's not just a once-off issue.
>> However, it is difficult to reproduce. I'm importing a lot of data into
>> a large repository and so far it's crashed on revision 5709 and 8409.
>>
>> I tried "bzr pack" today but it didn't cause it to crash again. The peak
>> memory usage during the pack was 479 MB, so it's unlikely to have really
>> run out of memory.
>>
>> I'm going to try it on a different machine and leave it running
>> overnight again.
>>
>> Is there a way to get a better stack trace from the crash? Would
>> debugging be easier if I built Bazaar from scratch instead of using the
>> redistributable?
>
> It could be. John, is there any way for him to build it with debug
> symbols included? Or indeed could we perhaps ship them in our
> standard build? It's been years and years since I dealt with msvc.
>

Passing args to the compiler is a bit tricky via setup.py. Specifically,
it doesn't seem to really have a way to say "if the compiler is X then
pass flag Y". You can tell it to pass a flag, but you have to figure out
what compiler is running manually, etc.

I believe the specific flag is "/Zi" which produces a .pdb file that
includes the debugging information. (Which is also compatible with all
levels of optimizations.)

If I just hack this in with:
=== modified file 'setup.py'
- --- setup.py 2009-11-02 21:51:25 +0000
+++ setup.py 2009-12-02 14:53:52 +0000
@@ -265,7 +265,9 @@
             source = [c_name]
     source.extend(extra_source)
     ext_modules.append(Extension(module_name, source,
- - define_macros=define_macros, libraries=libraries))
+ define_macros=define_macros, libraries=libraries,
+ extra_compile_args=['/Zi'],
+ extra_link_args=['/DEBUG']))

 add_pyrex_extension('bzrlib._annotator_pyx')

Then I do seem to get a .pdb file.

I'm attaching both the .pyd and the .pdb that is built this way. Perhaps
you can drop them into the same directory (overwriting the existing
ones) and see if it makes a difference?

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksWgOoACgkQJdeBCYSNAAM0UQCfcAeFCbVnDd1WVHhw3MzkN2Ua
ESMAn3xOuqjiyjy4fstprgcKYSfSe+LI
=mVrD
-----END PGP SIGNATURE-----

[Gareth White (gwhite-deactivatedaccount)]

Thanks John. Unfortunately those ones are built against Python 2.6 rather than 2.5 which the official Bazaar installation for Windows uses, so they don't work as drop-in replacements. I tried using your patch to rebuild the extensions against 2.5 but that requires Visual Studio 2003 which I don't think we have installed anywhere here!

If you're able to build a binary (plus pdb) of groupcompress from Bazaar 2.0.1 that's compatible with 2.5 I'll give it a go and see if I can get it to crash again. Alternatively I could try building all the extensions against 2.6 with Visual Studio 2005 and see if I can get that to crash. That's not really an ideal test though because it's changing too many variables at once.

It would be handy if debug symbols were built for every release of Bazaar and either included in the installer or provided as a separate zip file, as that would make debugging crashes like this a lot easier. I'll enter a bug for that.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Gareth White wrote:
> Thanks John. Unfortunately those ones are built against Python 2.6
> rather than 2.5 which the official Bazaar installation for Windows uses,
> so they don't work as drop-in replacements. I tried using your patch to
> rebuild the extensions against 2.5 but that requires Visual Studio 2003
> which I don't think we have installed anywhere here!
>
> If you're able to build a binary (plus pdb) of groupcompress from Bazaar
> 2.0.1 that's compatible with 2.5 I'll give it a go and see if I can get
> it to crash again. Alternatively I could try building all the extensions
> against 2.6 with Visual Studio 2005 and see if I can get that to crash.
> That's not really an ideal test though because it's changing too many
> variables at once.
>
> It would be handy if debug symbols were built for every release of
> Bazaar and either included in the installer or provided as a separate
> zip file, as that would make debugging crashes like this a lot easier.
> I'll enter a bug for that.
>

Well, the 2.5 builds are actually built using Mingw since 2003 is hard
to come by... Which means yet another set of debug flags. (I think it is
just -g, though, so I'll give it a try.)

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksX5FkACgkQJdeBCYSNAAOILQCfU08Tk2yudXSylPiSYonxOE8C
RMgAoMEtHaNNLt2ifd6T4bFhY3AdFzTs
=2/tB
-----END PGP SIGNATURE-----

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Gareth White wrote:
> Thanks John. Unfortunately those ones are built against Python 2.6
> rather than 2.5 which the official Bazaar installation for Windows uses,
> so they don't work as drop-in replacements. I tried using your patch to
> rebuild the extensions against 2.5 but that requires Visual Studio 2003
> which I don't think we have installed anywhere here!
>
> If you're able to build a binary (plus pdb) of groupcompress from Bazaar
> 2.0.1 that's compatible with 2.5 I'll give it a go and see if I can get
> it to crash again. Alternatively I could try building all the extensions
> against 2.6 with Visual Studio 2005 and see if I can get that to crash.
> That's not really an ideal test though because it's changing too many
> variables at once.
>
> It would be handy if debug symbols were built for every release of
> Bazaar and either included in the installer or provided as a separate
> zip file, as that would make debugging crashes like this a lot easier.
> I'll enter a bug for that.
>

I also found this link:

http://stackoverflow.com/questions/866721/how-to-generate-gcc-debug-symbol-outside-the-build-target

Which would be a way to always compile with debug flags, and then split
them into 2 parts. So that you can then download the debug information
later.

However, for what I'm doing, I'm just going to use -g.

On the flip side, I don't know that WinDBG knows anything about gcc
debug data. But it is at least worth a shot.

Anyway, attached is a python 2.5 version with -g enabled. You can try it
out and see if it helps.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksX5pIACgkQJdeBCYSNAAN4/QCgvJdarCHCuABkaryzKT0gvs5Y
t/0An1Ee63qdhL5N/Yihq6nsrtLLVe0G
=19nR
-----END PGP SIGNATURE-----

[Alexander Belchenko (bialix)]

John A Meinel пишет:
> Gareth White wrote:
>> Thanks John. Unfortunately those ones are built against Python 2.6
>> rather than 2.5 which the official Bazaar installation for Windows uses,
>> so they don't work as drop-in replacements. I tried using your patch to
>> rebuild the extensions against 2.5 but that requires Visual Studio 2003
>> which I don't think we have installed anywhere here!
>>
>> If you're able to build a binary (plus pdb) of groupcompress from Bazaar
>> 2.0.1 that's compatible with 2.5 I'll give it a go and see if I can get
>> it to crash again. Alternatively I could try building all the extensions
>> against 2.6 with Visual Studio 2005 and see if I can get that to crash.
>> That's not really an ideal test though because it's changing too many
>> variables at once.
>>
>> It would be handy if debug symbols were built for every release of
>> Bazaar and either included in the installer or provided as a separate
>> zip file, as that would make debugging crashes like this a lot easier.
>> I'll enter a bug for that.
>>
>
> I also found this link:
>
> http://stackoverflow.com/questions/866721/how-to-generate-gcc-debug-
> symbol-outside-the-build-target
>
> Which would be a way to always compile with debug flags, and then split
> them into 2 parts. So that you can then download the debug information
> later.
>
> However, for what I'm doing, I'm just going to use -g.
>
> On the flip side, I don't know that WinDBG knows anything about gcc
> debug data. But it is at least worth a shot.

I have MSVC 2003 and probably can try to build binaries with debug data.

__________ Information from ESET NOD32 Antivirus, version of virus signature database 4658 (20091203) __________

The message was checked by ESET NOD32 Antivirus.

http://www.esetnod32.ru

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander Belchenko wrote:
> John A Meinel пишет:
>> Gareth White wrote:
>>> Thanks John. Unfortunately those ones are built against Python 2.6
>>> rather than 2.5 which the official Bazaar installation for Windows uses,
>>> so they don't work as drop-in replacements. I tried using your patch to
>>> rebuild the extensions against 2.5 but that requires Visual Studio 2003
>>> which I don't think we have installed anywhere here!
>>>
>>> If you're able to build a binary (plus pdb) of groupcompress from Bazaar
>>> 2.0.1 that's compatible with 2.5 I'll give it a go and see if I can get
>>> it to crash again. Alternatively I could try building all the extensions
>>> against 2.6 with Visual Studio 2005 and see if I can get that to crash.
>>> That's not really an ideal test though because it's changing too many
>>> variables at once.
>>>
>>> It would be handy if debug symbols were built for every release of
>>> Bazaar and either included in the installer or provided as a separate
>>> zip file, as that would make debugging crashes like this a lot easier.
>>> I'll enter a bug for that.
>>>
>> I also found this link:
>>
>> http://stackoverflow.com/questions/866721/how-to-generate-gcc-debug-
>> symbol-outside-the-build-target
>>
>> Which would be a way to always compile with debug flags, and then split
>> them into 2 parts. So that you can then download the debug information
>> later.
>>
>> However, for what I'm doing, I'm just going to use -g.
>>
>> On the flip side, I don't know that WinDBG knows anything about gcc
>> debug data. But it is at least worth a shot.
>
> I have MSVC 2003 and probably can try to build binaries with debug data.
>

If you follow this thread, you should be able to make a 2-line change in
setup.py (extra_compile_args=["/Zi"], extra_link_args=["/DEBUG"]). and
then just run 'build', etc.

Thanks,
John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksX7KgACgkQJdeBCYSNAAMteQCgj6sqEKHzzSCeZmrXnJwadJXu
b5EAn2S2bGS0bun/tG2wpyKZt5YX05dR
=Lb1r
-----END PGP SIGNATURE-----

[Alexander Belchenko (bialix)]

Attached archive is result of running `setup.py build_ext` in bzr:2.0 branch. There is libs compiled without debug info (-nodebug) and with debug info (-debug). Also in the latter case vc70.pdb file was built, it's included as well.

I hope this helps.

[Gareth White (gwhite-deactivatedaccount)]

I had assumed the 2.5 builds were using VS2003 because the pyd files depended on msvcr71.dll. However, that must be due to some other reason. BTW, is there any reason why the standalone Bazaar installer uses 2.5 rather than 2.6?

In any case, I'm pretty sure WinDbg doesn't know how to interpret gcc debug info. I could probably use gdb but I'd have to reproduce the crash using the new pyd file.

I was able to force WinDbg to load Alex's pdb but I'm not sure the stack trace is correct. So I'm going to see if I can get it to crash again overnight using his pyd file. If not, I'll retry with gdb and John's one.

[Alexander Belchenko (bialix)]

    I had assumed the 2.5 builds were using VS2003 because the pyd files
depended on msvcr71.dll. However, that must be due to some other reason.
BTW, is there any reason why the standalone Bazaar installer uses 2.5
rather than 2.6?
Because it just works with 2.5.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Gareth White wrote:
> I had assumed the 2.5 builds were using VS2003 because the pyd files
> depended on msvcr71.dll. However, that must be due to some other reason.
> BTW, is there any reason why the standalone Bazaar installer uses 2.5
> rather than 2.6?
>

Mingw also knows how to depend on msvcr71.dll when building python
extensions. I think it even works for python2.6, but since there is
VS2008EE and it is currently publicly available, we use it to build the
2.6 extensions.

We use 2.5 because... we haven't updated yet. :)

Also, the last time I tried 2.6, it used an updated PyQt, and that
didn't play nicely with py2exe. (Specifically, PyQt had py3k
compatibility code, which py2exe then wanted to load. And when it tried,
it crashed because the syntax isn't valid py2.6 code. And if I taught
py2exe not to include that submodule, it would then fail saying that it
was missing a necessary submodule...)

> In any case, I'm pretty sure WinDbg doesn't know how to interpret gcc
> debug info. I could probably use gdb but I'd have to reproduce the crash
> using the new pyd file.
>
> I was able to force WinDbg to load Alex's pdb but I'm not sure the stack
> trace is correct. So I'm going to see if I can get it to crash again
> overnight using his pyd file. If not, I'll retry with gdb and John's
> one.
>

Can you include the stack trace just in case? It is also possible that
adding optimizations will cause the stack trace to be hard to interpret.
However, if it fails with both my .pyd and the one Alexander built, then
it is likely to be a genuine bug in the code, and hopefully any stack
trace will help us track it down.

I don't know if you've answered this, but do you have a clear way to
reproduce it? Is it possible for us to have access to that data?

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksdW+EACgkQJdeBCYSNAAMQrQCfZQXROUlWX/RNlGurSAFDPnIw
YmcAoLw7/3qlcPp0w58CvbxbgLNu8Te1
=+uj+
-----END PGP SIGNATURE-----

[Gareth White (gwhite-deactivatedaccount)]

It didn't crash last night using Bazaar 2.0.2 plus Alexander's pyd/pdb, but it wasn't reliably reproducible before so I'll keep trying.

I've attached the stack trace from the original crash dump, using Alexander's pdb file. I'm not completely sure it's correct because the pdb doesn't match the pyd file that crashed, but hopefully it'll help shed some light on the problem.

Unfortunately I don't really have a clear way to reproduce the crash yet. What we've been doing is running a tool to generate dummy data to get an idea of how Bazaar will scale if we decide to use it in production. When running this tool bzr.exe crashes (on average) every few thousand generated revisions. Since we're generating roughly 300 revisions an hour it takes some time to reproduce the crash! (It's faster if we don't generate merges but then the data isn't realistic.)

I can't share the repository because we used a recent snapshot of our (proprietary) codebase as a starting point. It's also quite large (1 GB). If we ran the tool on a dummy codebase to start with it and could still reproduce the problem then I could share both. I'm not sure whether the content of the files, the size of the files or even the number of files (22k) is significant.

Another option is to run the tool on a large public repository (e.g. MySQL) and see if it happens with that.

[Gareth White (gwhite-deactivatedaccount)]

I missed off some of the WinDbg output so I've updated the attachment.

[Martin Pool (mbp)]

You could try <http://bazaarvcs.wordpress.com/2009/10/28/mcrepogen/>
to generate test data.

--
Martin <http://launchpad.net/~mbp/>

[Gareth White (gwhite-deactivatedaccount)]

Thanks for the link. That does look like it could be quite useful...

For the dummy history we're generating we wanted to simulate multiple branches merging to/from a central branch. It doesn't look like mcrepogen does this yet, but perhaps it could be adapted.

[Neil Martinsen-Burrell (nmb)]

MCREPOGEN doesn't do merges between branches (yet). You could fairly easily embed its content-generating framework into your tool that simulated merges in your particular context.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

...
>
> Another option is to run the tool on a large public repository (e.g.
> MySQL) and see if it happens with that.
>

Well, I can say that converting the mysql codebase was one of my
benchmarks for the groupcompress code. So I'm sure that at least all of
their history doesn't trigger crashes (on my machine at least :).

I don't know what the difference would be with your generator, though.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksek0wACgkQJdeBCYSNAANxCACgj6y4Fi3Ieq6ogkBqXxd+MG3h
IdsAoM3I/RGz0ZmaJxn6ys995rVjY2RV
=6pIl
-----END PGP SIGNATURE-----

[Gareth White (gwhite-deactivatedaccount)]

After trying unsuccessfully for several days I managed to get it to crash again using Alexander's binaries. The stack trace is similar to the previous one but the line number for diff-delta.c looks more sensible this time.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Gareth White wrote:
> After trying unsuccessfully for several days I managed to get it to
> crash again using Alexander's binaries. The stack trace is similar to
> the previous one but the line number for diff-delta.c looks more
> sensible this time.
>
> ** Attachment added: "WinDbg analysis 3"
> http://launchpadlibrarian.net/36759299/crash3.txt
>

That does look like a better traceback.
        if (old_entry->ptr != NULL
            || old_entry >= old_index->hash[hash_offset + 1]) {

I originally thought we might have walked off the end of the hash
buckets, but looking with a bit more context:

    hash_offset = (entry->val & old_index->hash_mask);
    old_entry = old_index->hash[hash_offset + 1];
    old_entry--;
    while (old_entry->ptr == NULL
           && old_entry >= old_index->hash[hash_offset]) {
        old_entry--;
    }
    old_entry++;
    if (old_entry->ptr != NULL
        || old_entry >= old_index->hash[hash_offset + 1]) {

Rewriting it a bit...

    hash_offset = (entry->val & old_index->hash_mask);
    prev_entry = old_index->hash[hash_offset];
    next_entry = old_index->hash[hash_offset + 1];
    cur_entry = next_entry - 1;
    while (cur_entry->ptr == NULL && cur_entry >= prev_entry) {
      cur_entry--;
    }
    cur_entry++;
    if (cur_entry->ptr != NULL
        || cur_entry >= next_entry) {
    }

The structure is such that each bucket from the hash table references a
location in the data table and has some free space. The data table is
always arranged such that hash buckets are incremental. Something like:

hash_table |0 |1|2 |3|4
             | | | | |
entry_table abNNxycdefghijklmNNNN|

So this shows hash_offset 0 has 2 entries and two empty entries,
hash_offset 1 has two entries and no empty. offset 2 has a bunch of
entries, and 2 empties, and offset 3 has 2 empties. The hash array has
N+1 entries, and the last entry points to just past the last entry in
entry_table.

If we had something that fit into page 1, the above should start
pointing at 'c', then back up to y. it should see that cur_entry->ptr is
not NULL, and then increment back to 'c'. And see that this is both not
null, and == next_entry.

If it slotted into 3, we would grab the pointer from offset 4, if 3 were
then completely full, we would back up, see that 3 had no slots, then
increment back to the off-the-end location, and probably the
cur_entry->ptr is then going to fail, as cur_entry is pointing past the
end of entry_table, and thus "cur_entry->ptr" is not valid. Changing the
order of the if statement to:

if (cur_entry >= next_entry || cur_entry->ptr != NULL)

Should probably fix this.

John
=:->

  status inprogress
  assignee jameinel
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAksmWyUACgkQJdeBCYSNAAM2YACgwydWqgPGC1EMZRyju8R5bHus
pOIAoMsyYCIy1TuYXtebmWF6a7bz+B7b
=a5Ec
-----END PGP SIGNATURE-----

[John A Meinel (jameinel)]

The associated branch should have a fix for this. It certainly is a rare condition that is pretty hard to trigger. But I'm pretty sure I sorted out the logic of why it was happening.

Unfortunately, it is hard enough to trigger that I don't have a way to just say "run this" and it passes before, and fails after the fix. But regardless, I'm uploading a patch version of the .pyd file (built with mingw, maybe Alexander can do a VS2003 build?)

[Alexander Belchenko (bialix)]

Attached new pyd with debug info for MSVC 2003.

[Gareth White (gwhite-deactivatedaccount)]

Thanks John. I don't fully understand the hash table code but I can certainly see how your fix would avoid a crash in certain circumstances!

If I get a chance I'll see whether I can still reproduce a crash with the fixed pyd file. It's not likely I'll get a chance though with the Christmas holidays rapidly approaching.