segfault in dirstate process_entry under macports python

[Harish Narayanan]

I am having some difficulty getting bzr to run some kinds of commands (e.g. status, commit) on 32-bit Mac OS X 10.6.2. When I run such commands, I run into a segmentation fault.

$ bzr st
Bus error

The following is the section of the .bzr.log file corresponding to this error:

Tue 2010-02-16 19:28:08 +0000
0.070 bzr arguments: [u'st']
0.092 looking for plugins in /Users/harish/.bazaar/plugins
0.092 looking for plugins in /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/plugins
0.188 encoding stdout as sys.stdout encoding 'US-ASCII'
0.284 opening working tree '/Users/harish/Desktop/dorsal'
0.289 check paths: None

I receive the same error whether I install bzr by hand from source (2.0.4 and 2.1.0) or MacPorts.

Running python via gdb gives me the following backtrace:

#0 0x0162474e in __pyx_f_6bzrlib_21_dirstate_helpers_pyx_13ProcessEntryC__process_entry ()
#1 0x0161d09a in __pyx_f_6bzrlib_21_dirstate_helpers_pyx_13ProcessEntryC__iter_next ()
#2 0x01601cc3 in __pyx_f_6bzrlib_21_dirstate_helpers_pyx_13ProcessEntryC___next__ ()
#3 0x000b1f9d in PyEval_EvalFrameEx ()
#4 0x000b88af in PyEval_EvalCodeEx ()
#5 0x000b673c in PyEval_EvalFrameEx ()
#6 0x000b88af in PyEval_EvalCodeEx ()
#7 0x0003caf6 in function_call ()
#8 0x0000d195 in PyObject_Call ()
#9 0x000b49b0 in PyEval_EvalFrameEx ()
#10 0x000b88af in PyEval_EvalCodeEx ()
#11 0x000b673c in PyEval_EvalFrameEx ()
#12 0x000b88af in PyEval_EvalCodeEx ()
#13 0x000b673c in PyEval_EvalFrameEx ()
#14 0x000b88af in PyEval_EvalCodeEx ()
#15 0x000b673c in PyEval_EvalFrameEx ()
#16 0x000b88af in PyEval_EvalCodeEx ()
#17 0x0003caf6 in function_call ()
#18 0x0000d195 in PyObject_Call ()
#19 0x000b49b0 in PyEval_EvalFrameEx ()
#20 0x000b88af in PyEval_EvalCodeEx ()
#21 0x0003caf6 in function_call ()
#22 0x0000d195 in PyObject_Call ()
#23 0x000b49b0 in PyEval_EvalFrameEx ()
#24 0x000b88af in PyEval_EvalCodeEx ()
#25 0x0003c9fc in function_call ()
#26 0x0000d195 in PyObject_Call ()
#27 0x000b49b0 in PyEval_EvalFrameEx ()
#28 0x000b88af in PyEval_EvalCodeEx ()
#29 0x0003caf6 in function_call ()
#30 0x0000d195 in PyObject_Call ()
#31 0x000b49b0 in PyEval_EvalFrameEx ()
#32 0x000b88af in PyEval_EvalCodeEx ()
#33 0x000b673c in PyEval_EvalFrameEx ()
#34 0x000b6dc5 in PyEval_EvalFrameEx ()
#35 0x000b88af in PyEval_EvalCodeEx ()
#36 0x000b673c in PyEval_EvalFrameEx ()
#37 0x000b88af in PyEval_EvalCodeEx ()
#38 0x000b89b7 in PyEval_EvalCode ()
#39 0x000dd1ac in PyRun_FileExFlags ()
#40 0x000dd514 in PyRun_SimpleFileExFlags ()
#41 0x000ee2e0 in Py_Main ()
#42 0x00001f65 in ?? ()

When I run the self test, the following is what I see.

$ bzr --no-plugins selftest -v
running 0 tests...
testing: /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/bzr
   /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib
   bzr-2.0.4 python-2.6.4 Darwin-10.2.0-i386-32bit

   OK 82ms
.../Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/doc/api/transport.txt) OK 2ms
blackbox.test_add.TestAdd.test_add_control_dir(pre-views) OK 154ms
blackbox.test_add.TestAdd.test_add_control_dir(view-aware) OK 111ms
blackbox.test_add.TestAdd.test_add_dry_run(pre-views) OK 75ms
blackbox.test_add.TestAdd.test_add_dry_run(view-aware) OK 91ms
blackbox.test_add.TestAdd.test_add_from(pre-views) Segmentation fault

Also, I am able to run bzr branch and bzr pull. Can someone point me as to where to look?

[Gordon Tyler (doxxx)]

Which installer did you use?

[Harish Narayanan (hnarayanan)]

I tried both MacPorts, and compiling the source manually.

[Harish Narayanan (hnarayanan)]

This problem doesn't manifest when I use the system Python instead of the one provided by MacPorts. (I would like to use the MacPorts version, though.)

[Martin Pool (mbp)]

If you build from source with debug symbols can you find out which
line it's crashing on?

Have you made sure that the extensions were compiled against macports
python if you're running under it?

--
Martin <http://launchpad.net/~mbp/>

[Harish Narayanan (hnarayanan)]

1. I will try.

2. I went through the macports compilation messages and it appears that the extensions were compiled against macports python. e.g.,

/usr/bin/gcc-4.2 -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/opt/local/Library/Frameworks/Python.framework/Versions/2.6/include/python2.6 -c bzrlib/_dirstate_helpers_pyx.c -o build/temp.macosx-10.6-i386-2.6/bzrlib/_dirstate_helpers_pyx.o

[Harish Narayanan (hnarayanan)]

1. I think the following is the offending line:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x00000000
0x0162b74e in __pyx_f_6bzrlib_21_dirstate_helpers_pyx_13ProcessEntryC__process_entry (__pyx_v_self=0x17b8e90, __pyx_v_entry=0x188e0a8, __pyx_v_path_info=0x1897e40) at bzrlib/_dirstate_helpers_pyx.c:5225
5225 Py_DECREF(__pyx_v_old_path_u);

And here is the updated backtrace:

#0 0x0162b74e in __pyx_f_6bzrlib_21_dirstate_helpers_pyx_13ProcessEntryC__process_entry (__pyx_v_self=0x17b8e90, __pyx_v_entry=0x188e0a8, __pyx_v_path_info=0x1897e40) at bzrlib/_dirstate_helpers_pyx.c:5225
#1 0x0162409a in __pyx_f_6bzrlib_21_dirstate_helpers_pyx_13ProcessEntryC__iter_next (__pyx_v_self=0x17b8e90) at bzrlib/_dirstate_helpers_pyx.c:5850
#2 0x01608cc3 in __pyx_f_6bzrlib_21_dirstate_helpers_pyx_13ProcessEntryC___next__ (__pyx_v_self=0x17b8e90) at bzrlib/_dirstate_helpers_pyx.c:5472
#3 0x000b1f9d in PyEval_EvalFrameEx ()
#4 0x000b88af in PyEval_EvalCodeEx ()
#5 0x000b673c in PyEval_EvalFrameEx ()
#6 0x000b88af in PyEval_EvalCodeEx ()
#7 0x0003caf6 in function_call ()
#8 0x0000d195 in PyObject_Call ()
#9 0x000b49b0 in PyEval_EvalFrameEx ()
#10 0x000b88af in PyEval_EvalCodeEx ()
#11 0x000b673c in PyEval_EvalFrameEx ()
#12 0x000b88af in PyEval_EvalCodeEx ()
#13 0x000b673c in PyEval_EvalFrameEx ()
#14 0x000b88af in PyEval_EvalCodeEx ()
#15 0x000b673c in PyEval_EvalFrameEx ()
#16 0x000b88af in PyEval_EvalCodeEx ()
#17 0x0003caf6 in function_call ()
#18 0x0000d195 in PyObject_Call ()
#19 0x000b49b0 in PyEval_EvalFrameEx ()
#20 0x000b88af in PyEval_EvalCodeEx ()
#21 0x0003caf6 in function_call ()
#22 0x0000d195 in PyObject_Call ()
#23 0x000b49b0 in PyEval_EvalFrameEx ()
#24 0x000b88af in PyEval_EvalCodeEx ()
#25 0x0003caf6 in function_call ()
#26 0x0000d195 in PyObject_Call ()
#27 0x000b49b0 in PyEval_EvalFrameEx ()
#28 0x000b88af in PyEval_EvalCodeEx ()
#29 0x0003caf6 in function_call ()
#30 0x0000d195 in PyObject_Call ()
#31 0x000b49b0 in PyEval_EvalFrameEx ()
#32 0x000b88af in PyEval_EvalCodeEx ()
#33 0x0003caf6 in function_call ()
#34 0x0000d195 in PyObject_Call ()
#35 0x000b49b0 in PyEval_EvalFrameEx ()
#36 0x000b88af in PyEval_EvalCodeEx ()
#37 0x0003c9fc in function_call ()
#38 0x0000d195 in PyObject_Call ()
#39 0x000b49b0 in PyEval_EvalFrameEx ()
#40 0x000b88af in PyEval_EvalCodeEx ()
#41 0x0003caf6 in function_call ()
#42 0x0000d195 in PyObject_Call ()
#43 0x000b49b0 in PyEval_EvalFrameEx ()
#44 0x000b88af in PyEval_EvalCodeEx ()
#45 0x000b673c in PyEval_EvalFrameEx ()
#46 0x000b6dc5 in PyEval_EvalFrameEx ()
#47 0x000b88af in PyEval_EvalCodeEx ()
#48 0x000b673c in PyEval_EvalFrameEx ()
#49 0x000b88af in PyEval_EvalCodeEx ()
#50 0x000b89b7 in PyEval_EvalCode ()
#51 0x000dd1ac in PyRun_FileExFlags ()
#52 0x000dd514 in PyRun_SimpleFileExFlags ()
#53 0x000ee2e0 in Py_Main ()
#54 0x00001f65 in ?? ()

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Harish Narayanan wrote:
> 1. I think the following is the offending line:
>
> Program received signal EXC_BAD_ACCESS, Could not access memory.
> Reason: KERN_PROTECTION_FAILURE at address: 0x00000000
> 0x0162b74e in __pyx_f_6bzrlib_21_dirstate_helpers_pyx_13ProcessEntryC__process_entry (__pyx_v_self=0x17b8e90, __pyx_v_entry=0x188e0a8, __pyx_v_path_info=0x1897e40) at bzrlib/_dirstate_helpers_pyx.c:5225
> 5225 Py_DECREF(__pyx_v_old_path_u);
>

#1 Do we know what version of Pyrex was used to generate the .c file? It
is possible there is a bug we should track down. (If you are using the
tarball, then it should be pyrex 0.9.8.5, but we'd see a problem on
other platforms then.)

#2 For example, pyrex 0.9.4.* had a known bug where it would use
Py_DECREF instead of Py_XDECREF and that would cause problems with
trying to decrement a NULL pointer.

Alternatively we could look ourselves and see if there is an unhandled
code path that is causing 'old_path_u' to be set to NULL.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkt9V70ACgkQJdeBCYSNAAORXwCcDmaH4K45n7Ejp/hX7/Eb5nYy
ZO0An3ERsAE+jTFw80kZqMZxL6tY5+XA
=eafi
-----END PGP SIGNATURE-----

[Harish Narayanan (hnarayanan)]

Also, it appears to work when I comment out the following lines (5225--5227) in bzrlib/_dirstate_helpers_pyx.c. I don't know of the implications of doing so, as I am not able to understand the pyrex generated code.

/* Py_DECREF(__pyx_v_old_path_u); */
/* Py_DECREF(__pyx_v_path_u); */
/* Py_DECREF(__pyx_v_source_kind); */

[Harish Narayanan (hnarayanan)]

I am using the 2.1.0 tarball. On top of this pyrex-generated file, it says the following:
/* Generated by Pyrex 0.9.8.5 on Thu Jan 21 14:10:44 2010 */

[Martin Pool (mbp)]

On 19 February 2010 02:18, Harish Narayanan <email address hidden> wrote:
> I am using the 2.1.0 tarball. On top of this pyrex-generated file, it says the following:
> /* Generated by Pyrex 0.9.8.5 on Thu Jan 21 14:10:44 2010 */

Thanks, could you please paste a larger section around the lines in
question, including the bit that shows which pyrex lines it came from?

--
Martin <http://launchpad.net/~mbp/>

[Harish Narayanan (hnarayanan)]

  /* "/home/jameinel/dev/bzr/work/bzrlib/_dirstate_helpers_pyx.pyx":1369 */
  __pyx_12 = PyTuple_New(2); if (!__pyx_12) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 1369; goto __pyx_L1;}
  Py_INCREF(Py_None);
  PyTuple_SET_ITEM(__pyx_12, 0, Py_None);
  Py_INCREF(Py_None);
  PyTuple_SET_ITEM(__pyx_12, 1, Py_None);
  __pyx_r = __pyx_12;
  __pyx_12 = 0;
  goto __pyx_L0;

  __pyx_r = Py_None; Py_INCREF(Py_None);
  goto __pyx_L0;
  __pyx_L1:;
  Py_XDECREF(__pyx_1);
  Py_XDECREF(__pyx_3);
  Py_XDECREF(__pyx_5);
  Py_XDECREF(__pyx_7);
  Py_XDECREF(__pyx_8);
  Py_XDECREF(__pyx_9);
  Py_XDECREF(__pyx_10);
  Py_XDECREF(__pyx_11);
  Py_XDECREF(__pyx_12);
  __Pyx_AddTraceback("bzrlib._dirstate_helpers_pyx.ProcessEntryC._process_entry");
  __pyx_r = 0;
  __pyx_L0:;
  Py_DECREF(__pyx_v_file_id);
  Py_DECREF(__pyx_v_details_list);
  Py_DECREF(__pyx_v_source_details);
  Py_DECREF(__pyx_v_target_details);
  Py_DECREF(__pyx_v_link_or_sha1);
  Py_DECREF(__pyx_v_old_dirname);
  Py_DECREF(__pyx_v_old_basename);
  Py_DECREF(__pyx_v_old_path);
  Py_DECREF(__pyx_v_path);
  Py_DECREF(__pyx_v_old_entry);
  Py_DECREF(__pyx_v_target_kind);
  Py_DECREF(__pyx_v_target_exec);
  Py_DECREF(__pyx_v_statvalue);
  Py_DECREF(__pyx_v_source_parent_id);
  Py_DECREF(__pyx_v_source_parent_entry);
  Py_DECREF(__pyx_v_new_dirname);
  Py_DECREF(__pyx_v_target_parent_id);
  Py_DECREF(__pyx_v_target_parent_entry);
  Py_DECREF(__pyx_v_source_exec);
  Py_DECREF(__pyx_v_changed);
  Py_DECREF(__pyx_v_old_path_u); //1. These are the troublesome lines
  Py_DECREF(__pyx_v_path_u); // 2. These are the troublesome lines
  Py_DECREF(__pyx_v_source_kind); // 3. These are the troublesome lines
  Py_DECREF(__pyx_v_parent_entry);
  Py_DECREF(__pyx_v_parent_id);
  Py_DECREF(__pyx_v_self);
  Py_DECREF(__pyx_v_entry);
  Py_DECREF(__pyx_v_path_info);
  return __pyx_r;
}

[Martin Pool (mbp)]

Thanks.

So as you might expect from the C code, this is the
         return None, None
at the end of _process_entry, where it's deferencing everything that was in the C frame.

Re comment #8, do you need to comment out all three of those lines to make it work?

I suspect that in about line 1265-1277 a Python error is being raised and we don't notice it, therefore the variable continues set to null and we crash on the decref. I'm not sure why Pyrex isn't automatically checking.

[Martin Pool (mbp)]

So in my built C source, it certainly does seem to have checks there.

I wonder if this is a bug in the Pyrex used by Macports. What version is it?

[Harish Narayanan (hnarayanan)]

I needed to comment out each of the three lines to make it work. (I tried many combinations systematically.)

MacPorts has Pyrex 0.9.8.5.

[Harish Narayanan (hnarayanan)]

Could someone point me on how I should try to debug this? I am having difficulty reading Pyrex code.

Should I be looking for null/undefined variables in the .pyx file corresponding to these offending lines?

[Martin Pool (mbp)]

On 25 February 2010 21:53, Harish Narayanan <email address hidden> wrote:
> Could someone point me on how I should try to debug this? I am having
> difficulty reading Pyrex code.
>
> Should I be looking for null/undefined variables in the .pyx file
> corresponding to these offending lines?

Can you attach the generated C file to this bug?

We probably need to be looking in the C code for cases where it can
reach those lines with the variables set to null. I think this is at
a lower level than the pyrex source.

--
Martin <http://launchpad.net/~mbp/>

[Harish Narayanan (hnarayanan)]

I meant to say pyrex-generated source. :) And I have attached it.

This version is regenerated on my machine (during a manual install) using Pyrex 0.98.5 (from MacPorts). It shows the same problem (except on lines 5232--5234).

[Harish Narayanan (hnarayanan)]

Also, I think the problem might be elsewhere. I found another (64-bit) machine with MacPorts and Snow Leopard and tried to build bzr on it. It worked out of the box. I did a diff between the two pyrex-generated files and they are identical.

--- _dirstate_helpers_pyx.c 2010-03-04 14:10:58.000000000 +0100
+++ bzr-2.1.0/bzrlib/_dirstate_helpers_pyx.c 2010-03-04 14:20:06.000000000 +0100
@@ -1,4 +1,4 @@
-/* Generated by Pyrex 0.9.8.5 on Thu Mar 4 14:08:11 2010 */
+/* Generated by Pyrex 0.9.8.5 on Thu Mar 4 14:20:06 2010 */

 #define PY_SSIZE_T_CLEAN
 #include "Python.h"