[os_keystone] uwsgi process segfaults on openSUSE

[Markos Chandras]

Hi,

the openSUSE job (via vagrant) started to fail recently with the following backtrace in /var/log/keystone/keystone-wsgi-public.log and /var/log/keystone/keystone-wsgi-admin.log

*** Starting uWSGI 2.0.15 (64bit) on [Thu Jul 20 16:41:28 2017] ***
compiled with version: 4.8.5 on 20 July 2017 16:38:12
os: Linux-4.4.57-18.3-default #1 SMP Thu Mar 30 06:39:47 UTC 2017 (39c8557)
nodename: keystone1
machine: x86_64
clock source: unix
detected number of CPU cores: 2
current working directory: /
detected binary path: /openstack/venvs/keystone-testing/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your processes number limit is 1850
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: enabled
uWSGI http bound on :37359 fd 3
uwsgi socket 0 bound to TCP address 127.0.0.1:5001 fd 6
Python version: 2.7.13 (default, Mar 22 2017, 12:31:17) [GCC]
Set PythonHome to /openstack/venvs/keystone-testing
Python main interpreter initialized at 0x1c47670
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 402621 bytes (393 KB) for 2 cores
*** Operational MODE: preforking ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 18180)
spawned uWSGI worker 1 (pid: 18183, cores: 1)
spawned uWSGI worker 2 (pid: 18184, cores: 1)
spawned uWSGI http 1 (pid: 18185)
!!! uWSGI process 18184 got Segmentation Fault !!!
*** backtrace of 18184 ***
/openstack/venvs/keystone-testing/bin/uwsgi(uwsgi_backtrace+0x2e) [0x46bd7e]
/openstack/venvs/keystone-testing/bin/uwsgi(uwsgi_segfault+0x21) [0x46c111]
/lib64/libc.so.6(+0x34950) [0x7f7e9d2f2950]
/lib64/libc.so.6(+0x90e5a) [0x7f7e9d34ee5a]
/lib64/libcrypto.so.1.0.0(+0x1201c9) [0x7f7e9e5621c9]
/lib64/libcrypto.so.1.0.0(lh_insert+0x42) [0x7f7e9e5624a2]
/lib64/libcrypto.so.1.0.0(OBJ_NAME_add+0x6f) [0x7f7e9e4b28ff]
/openstack/venvs/keystone-testing/lib/python2.7/site-packages/cryptography/hazmat/bindings/../../.libs/libssl-abb4988e.so.1.1(+0x2cbb5) [0x7f7e979debb5]
/lib64/libpthread.so.0(+0x6c13) [0x7f7e9f225c13]
/openstack/venvs/keystone-testing/lib/python2.7/site-packages/cryptography/hazmat/bindings/../../.libs/libcrypto-2ae7ec4c.so.1.1(CRYPTO_THREAD_run_once+0x9) [0x7f7e976c6399]
/openstack/venvs/keystone-testing/lib/python2.7/site-packages/cryptography/hazmat/bindings/../../.libs/libssl-abb4988e.so.1.1(OPENSSL_init_ssl+0x73) [0x7f7e979ded43]
/openstack/venvs/keystone-testing/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so(+0x5a51d) [0x7f7e97c8151d]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x5547) [0x7f7e9d98dac7]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x24c) [0x7f7e9d99330c]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x59d0) [0x7f7e9d98df50]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x24c) [0x7f7e9d99330c]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x59d0) [0x7f7e9d98df50]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x24c) [0x7f7e9d99330c]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f7e9d9e8252]
/usr/lib64/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xb0) [0x7f7e9d9ec940]
/usr/lib64/libpython2.7.so.1.0(+0x150b5c) [0x7f7e9d9ecb5c]
/usr/lib64/libpython2.7.so.1.0(+0x10bdc4) [0x7f7e9d9a7dc4]
/usr/lib64/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x483) [0x7f7e9d9a84b3]
/usr/lib64/libpython2.7.so.1.0(+0xeb8cb) [0x7f7e9d9878cb]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46) [0x7f7e9d91f986]
/usr/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x36) [0x7f7e9d988096]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3310) [0x7f7e9d98b890]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x24c) [0x7f7e9d99330c]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f7e9d9e8252]
/usr/lib64/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xb0) [0x7f7e9d9ec940]
/usr/lib64/libpython2.7.so.1.0(+0x150b5c) [0x7f7e9d9ecb5c]
/usr/lib64/libpython2.7.so.1.0(+0x10bdc4) [0x7f7e9d9a7dc4]
/usr/lib64/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x3c7) [0x7f7e9d9a83f7]
/usr/lib64/libpython2.7.so.1.0(+0xeb8cb) [0x7f7e9d9878cb]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46) [0x7f7e9d91f986]
/usr/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x36) [0x7f7e9d988096]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3310) [0x7f7e9d98b890]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x24c) [0x7f7e9d99330c]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f7e9d9e8252]
/usr/lib64/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xb0) [0x7f7e9d9ec940]
/usr/lib64/libpython2.7.so.1.0(+0x150b5c) [0x7f7e9d9ecb5c]
/usr/lib64/libpython2.7.so.1.0(+0x1511d3) [0x7f7e9d9ed1d3]
/usr/lib64/libpython2.7.so.1.0(+0x10ba10) [0x7f7e9d9a7a10]
/usr/lib64/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x32e) [0x7f7e9d9a835e]
/usr/lib64/libpython2.7.so.1.0(+0xeb8cb) [0x7f7e9d9878cb]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46) [0x7f7e9d91f986]
/usr/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x36) [0x7f7e9d988096]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3310) [0x7f7e9d98b890]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x24c) [0x7f7e9d99330c]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f7e9d9e8252]
/usr/lib64/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xb0) [0x7f7e9d9ec940]
/usr/lib64/libpython2.7.so.1.0(+0x150b5c) [0x7f7e9d9ecb5c]
/usr/lib64/libpython2.7.so.1.0(+0x6b249) [0x7f7e9d907249]
/usr/lib64/libpython2.7.so.1.0(+0x1511d3) [0x7f7e9d9ed1d3]
/usr/lib64/libpython2.7.so.1.0(+0x10bc02) [0x7f7e9d9a7c02]
/usr/lib64/libpython2.7.so.1.0(PyImport_ImportModuleLevel+0x2b7) [0x7f7e9d9a82e7]
/usr/lib64/libpython2.7.so.1.0(+0xeb8cb) [0x7f7e9d9878cb]
/usr/lib64/libpython2.7.so.1.0(PyObject_Call+0x46) [0x7f7e9d91f986]
/usr/lib64/libpython2.7.so.1.0(PyEval_CallObjectWithKeywords+0x36) [0x7f7e9d988096]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3310) [0x7f7e9d98b890]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x24c) [0x7f7e9d99330c]
/usr/lib64/libpython2.7.so.1.0(PyEval_EvalCode+0x32) [0x7f7e9d9e8252]
/usr/lib64/libpython2.7.so.1.0(PyImport_ExecCodeModuleEx+0xb0) [0x7f7e9d9ec940]
/usr/lib64/libpython2.7.so.1.0(+0x150b5c) [0x7f7e9d9ecb5c]
*** end of backtrace ***

The vagrant image is the same and nothing really changed in the os_keystone lately that would justify this breakage

[Markos Chandras (hwoarang)]

Reverting back to cryptography 1.9 fixes the problem

upstream bug reported at https://github.com/pyca/cryptography/issues/3804

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/486173

[Markos Chandras (hwoarang)]

Got a few more information from Dirk Mueller on the #openstack-infra channel this morning

[08:55:00] <dirk> hwoarang: yes, because cryptography 2.0 added a universal wheel with an embedded openssl 1.1
[08:55:00] <dirk> which has symbol clashes with the opensuse provided openssl 1.0
[08:55:00] <dirk> it doesn't happen on any other distro in infra because infra is rebuilding all wheels and provides a wheel mirror
[08:55:15] <dirk> https://review.openstack.org/#/c/486305/ is going to fix that
[08:55:25] <dirk> but its actually a terrible workaround
[08:55:31] <dirk> hwoarang: the --no-binary is imho more correct

[OpenStack Infra (hudson-openstack)]

Change abandoned by Markos Chandras (hwoarang) (<email address hidden>) on branch: master
Review: https://review.openstack.org/486173
Reason: The problem appears to be with the wheel package so abandoning this

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/486580

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/486656

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/486580
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-tests/commit/?id=321123b018f20f774e697d87f4f55b9cb78bba6f
Submitter: Jenkins
Branch: master

commit 321123b018f20f774e697d87f4f55b9cb78bba6f
Author: Markos Chandras <email address hidden>
Date: Mon Jul 24 12:33:55 2017 +0100

    test-vars: Always build cryptography from source

    cryptography may bundle openssl in the wheel and that causes symbol
    conflicts if a different openssl is provided by the distribution.
    As such, it's probably safer to re-build cryptography ourselves just
    to be sure that the correct distro libraries are used. We may want to
    revert that once we start building wheel packages for openSUSE and
    distribute them in the OpenStack mirrors since every distribution
    well then get a proper wheel file for its needs. See related review
    https://review.openstack.org/#/c/486305/

    Closes-Bug: 1705521
    Link: https://github.com/pyca/cryptography/issues/3804
    Change-Id: I7e88935acda580d8522a1e6927ea498431d78bda

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/486173
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=9220732958c2d0303e5a4b77a484a412ef6d605b
Submitter: Jenkins
Branch: master

commit 9220732958c2d0303e5a4b77a484a412ef6d605b
Author: Markos Chandras <email address hidden>
Date: Fri Jul 21 17:07:00 2017 +0100

    group_vars: repo_all: Always build cryptography from source

    cryptography may bundle openssl in the wheel and that causes symbol
    conflicts if a different openssl is provided by the distribution.
    As such, it's probably safer to re-build cryptography ourselves just
    to be sure that the correct distro libraries are used. This has been
    addressed in openstack-ansible-tests/test-vars.yaml
    (https://review.openstack.org/#/c/486580/) to fix the CI tests but the
    problem is also present on regular deployments so we set it in the
    group_variables for the repo_all group of hosts so it's built from
    source in the wheel repository.

    Related-Bug: 1705521
    Link: https://github.com/pyca/cryptography/issues/3804
    Change-Id: I54ba3c1fa48a2f4c633930bc7e8cc65397f86659

[Matthew Thode (prometheanfire)]

Does this bug still apply to cryptography-2.0.2?

[Markos Chandras (hwoarang)]

Probably not but I haven't tested it. In any case, I would say that we should still build it from source since it seems a rather fragile situation.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Markos Chandras (hwoarang) (<email address hidden>) on branch: master
Review: https://review.openstack.org/486656