python2.7 crashes with SegFault when running Zope

[Ján Jockusch]

Source package / Package:

python2.7_2.7.11-7ubuntu1

System:

Description: Ubuntu 16.04 LTS
Release: 16.04

This is a bug report originating from the findings documented in

https://github.com/zopefoundation/Zope/issues/56

The expected behaviour is this:
- Upon installing Zope2 in a virtualenv a default instance should work with no issues.

What happens instead:
- The freshly installed Zope instance causes Python to SegFault after only a few requests.

But Python should never SegFault.

The following program produces the error:

#!/bin/bash
set -e

# Call this testing script with an argument "setup" for the first time
# to build the testing case
#
# After that, the script only performs the test as described below.
#
# Requirements: apt-get install virtualenv python2.7-dbg
#
# This script needs internet access when running.

# The python version to test:
python=/usr/bin/python2.7-dbg

# This part builds the testing environment

if [ "$1" == "setup" ] ; then
    # Virtual environment with the chosen python
    virtualenv --python ${python} --no-site-packages zope-virtualenv
    cd zope-virtualenv
    # Basic Zope installation
    ./bin/easy_install -i http://download.zope.org/Zope2/index/2.13.24 Zope2
    # Simplest possible Zope instance
    ./bin/mkzopeinstance --dir instance --user admin:admin
    cd ..
fi

# THE TEST

# Run Zope instance in the background
zope-virtualenv/instance/bin/runzope &

# Give Zope some time to start up
sleep 3

# Call a few management URLs, nothing too fancy...
urls="
http://admin:admin@localhost:8080/Control_Panel/manage_main
http://admin:admin@localhost:8080/Control_Panel/Database/main/manage_main
http://admin:admin@localhost:8080/Control_Panel/Database/manage_main
http://admin:admin@localhost:8080/Control_Panel/Database/main/manage_activity
http://admin:admin@localhost:8080/manage
http://admin:admin@localhost:8080/acl_users/manage_main
http://admin:admin@localhost:8080/acl_users/manage_access
"

while true ; do
    for url in ${urls} ; do
 echo ${url}
        curl -s $url -o /dev/null
    done
done

# This test fails after about 10 requests with
# Fatal Python error: ../Objects/descrobject.c:10 object at 0x??? has negative ref count -1

[Ján Jockusch (jan-jockusch)]

Important additional information:

Stock pythons from python.org do not exhibit the problem.

Tested with:
- Python 2.7.9
- Python 2.7.10
- Python 2.7.11

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Matthias Klose (doko)]

please report this upstream, or else you'll see that in 2.7.12 upstream.

[Patrick Gerken (do3cc)]

Hi Matthias, I am not sure I understand what you mean by upstream. As you wrote this also in the github tracker, I guess you mean to Python directly.
Several Plone developers tested Plone independently with a stock, self compiled Python 2.7.11. and confirmed, that their Plone/Zope instance does not segfault. So the patches in Ubuntu Python must be the source of the problem.

[Alexander Loechel (alexander-loechel)]

Hi Patrick,

I think we need to verify that this is in Ubuntu itself, while Ubuntu just uses the upstream Package from Debian (https://packages.debian.org/search?keywords=python2.7 / https://packages.debian.org/stretch/python2.7 or https://packages.debian.org/sid/python2.7) if they are also affected. If so, we really need to go upstream for a bug Report on Debian Python 2.7

[Ján Jockusch (jan-jockusch)]

The Debian (testing) version of Python2.7.11 does also exhibit the problem. I already told the Debian maintainer, Matthias Klose <email address hidden>, via email because I wanted to know exactly where he prefers to have it posted. He has already contributed here.

@Matthias: The issue remains open. Please advise on how I should proceed concerning Debian.

In the mean time, I have attempted to narrow the debian/patches/* files down so I can find which patch produces the problem.

But the patches quite heavily depend on each other, so I have not yet succeeded in producing a debianized build which does not have the problem. Still working at that.

The bug can be traced in gdb by executing the testing setup, then saying "gdb ./zope-virtualenv/bin/python2.7-dbg" and in gdb: "run ./zope-virtualenv/bin/runzope -C ./zope-virtualenv/instance/etc/zope.conf".

After sending the curl commands to produce the error, gdb exits with the above mentioned error:

"Fatal Python error: ../Objects/descrobject.c:10 object at 0x1572240 has negative ref count -1
Program received signal SIGABRT, Aborted."

Then the stack trace can be inspected. My knowledge of Python's memory management is not deep enough to really make any sense of what I'm seeing, other than it looks remotely like a "double free".

I'll try to contribute more. Anyone with better understanding of Python internals, your help is much appreciated!

[Matthias Klose (doko)]

On 17.05.2016 14:14, Ján Jockusch wrote:
> In the mean time, I have attempted to narrow the debian/patches/* files
> down so I can find which patch produces the problem.

it should be enough to disable the hg-updates patch.

[Ján Jockusch (jan-jockusch)]

Yes, that was exactly the one I wanted to keep out. Sadly, it's not that
simple. Later patches depend on this one. If I also disable those, the
build fails. That's why it's taking so much time...

On 05/24/2016 10:14 AM, Matthias Klose wrote:
> On 17.05.2016 14:14, Ján Jockusch wrote:
>> In the mean time, I have attempted to narrow the debian/patches/* files
>> down so I can find which patch produces the problem.
> it should be enough to disable the hg-updates patch.
>

[Matthias Klose (doko)]

looks to be fixed in yakkety

[Ján Jockusch (jan-jockusch)]

Confirm that the problem is fixed in yakkety.

I just repeated the test by doing the following on a Ubuntu16.04 installation:

# dpkg -i python2.7-minimal_2.7.11-11_amd64.deb libpython2.7-minimal_2.7.11-11_amd64.deb
Preparing to unpack python2.7-minimal_2.7.11-11_amd64.deb ...
Unpacking python2.7-minimal (2.7.11-11) over (2.7.11-11) ...
Preparing to unpack libpython2.7-minimal_2.7.11-11_amd64.deb ...
Unpacking libpython2.7-minimal:amd64 (2.7.11-11) over (2.7.11-7ubuntu1) ...
Setting up libpython2.7-minimal:amd64 (2.7.11-11) ...
Setting up python2.7-minimal (2.7.11-11) ...

After raising the version of the python2.7 executable in this fashion from 2.7.11-7ubuntu1 to 2.7.11-11 I repeated the testing sequence with the script I posted earlier, with no problem.

Is there a recommended go-ahead as to which packages I should upgrade?

Will the patch move into 16.04 LTS updates?

[Jens KLEIN (jensens)]

Any update if this fix will go into 16.04 LTS?

[Jens KLEIN (jensens)]

Sorry for impatience. I just found it is already in proposed updates.

[Matthias Klose (doko)]

fixed in 16.04.01 LTS and newer releases