| 2019-01-29 09:20:53 |
James Page |
description |
Running $ gnocchi status --debug
REQ: curl -g -i -X GET http://10.5.0.38:5000/v3 -H "Accept: application/json" -H "User-Agent: gnocchi keystoneauth1/3.10.0 python-requests/2.18.4 CPython/3.6.7"
Starting new HTTP connection (1): 10.5.0.38
http://10.5.0.38:5000 "GET /v3 HTTP/1.1" 200 249
RESP: [200] Connection: Keep-Alive Content-Length: 249 Content-Type: application/json Date: Mon, 28 Jan 2019 11:34:39 GMT Keep-Alive: timeout=5, max=100 Server: Apache/2.4.29 (Ubuntu) Vary: X-Auth-Token X-Distribution: Ubuntu x-openstack-request-id: req-d74445b8-1567-41d0-b741-a27000a80f14
RESP BODY: {"version": {"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "http://10.5.0.38:5000/v3/", "rel": "self"}]}}
GET call to http://10.5.0.38:5000/v3 used request id req-d74445b8-1567-41d0-b741-a27000a80f14
Making authentication request to http://10.5.0.38:5000/v3/auth/tokens
http://10.5.0.38:5000 "POST /v3/auth/tokens HTTP/1.1" 201 1780
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "5a1b23e27bbe47609df2ed5aae809543", "name": "Admin"}, {"id": "9499c375bc694f55891f5c2ed5b89a90", "name": "Member"}], "is_admin_project": true, "project": {"domain": {"id": "d008601304584178a632480f1b5658f1", "name": "admin_domain"}, "id": "efd4af8ba3c147379f140e0fcf03c1a0", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://10.5.0.38:35357/v3", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "564d822b8912406ca8ca86ff0e6fdde3"}, {"url": "http://10.5.0.38:5000/v3", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "56e351a2cff74c24beebf951cea8f099"}, {"url": "http://10.5.0.38:5000/v3", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "f451dcf182b14bdd9955f4d526011bf9"}], "type": "identity", "id": "f4528ee1a55e434488244066eae3bb70", "name": "keystone"}, {"endpoints": [{"url": "http://10.5.0.17:8041", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "93b7789c73f24134971a99cf9773e45c"}, {"url": "http://10.5.0.17:8041", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "e2c3aea790d54df89e4c36b815ce0f18"}, {"url": "http://10.5.0.17:8041", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f20ba6de8db8453bbd57cfc9f0e18e51"}], "type": "metric", "id": "39131ce26e2a40f18b56a89f61d76bce", "name": "gnocchi"}], "expires_at": "2019-01-28T12:34:39.000000Z", "user": {"password_expires_at": null, "domain": {"id": "d008601304584178a632480f1b5658f1", "name": "admin_domain"}, "id": "f656e0c9ff90432ca6a16fbcf689c39b", "name": "admin"}, "audit_ids": ["ORciQMnASuaRQBdBa6uuIg"], "issued_at": "2019-01-28T11:34:39.000000Z"}}
REQ: curl -g -i -X GET http://10.5.0.17:8041/v1/status?details=False -H "Accept: application/json, */*" -H "User-Agent: gnocchi keystoneauth1/3.10.0 python-requests/2.18.4 CPython/3.6.7" -H "X-Auth-Token: {SHA1}f16e953e1742980359603aef5a704611dd23a651"
Starting new HTTP connection (1): 10.5.0.17
http://10.5.0.17:8041 "GET /v1/status?details=False HTTP/1.1" 500 608
RESP: [500] Connection: close Content-Length: 608 Content-Type: text/html; charset=iso-8859-1 Date: Mon, 28 Jan 2019 11:34:39 GMT Server: Apache/2.4.29 (Ubuntu)
RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.17 Port 8041</address>
</body></html>
(HTTP 500)
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python3/dist-packages/cliff/display.py", line 116, in run
column_names, data = self.take_action(parsed_args)
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status_cli.py", line 23, in take_action
status = utils.get_client(self).status.get()
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status.py", line 21, in get
return self._get(self.url + '?details=%s' % details).json()
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py", line 37, in _get
return self.client.api.get(*args, **kwargs)
File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 328, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python3/dist-packages/gnocchiclient/client.py", line 52, in request
raise exceptions.from_response(resp, method)
gnocchiclient.exceptions.ClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.17 Port 8041</address>
</body></html>
(HTTP 500)
Traceback (most recent call last):
File "/usr/bin/gnocchi", line 10, in <module>
sys.exit(main())
File "/usr/lib/python3/dist-packages/gnocchiclient/shell.py", line 251, in main
return GnocchiShell().run(args)
File "/usr/lib/python3/dist-packages/cliff/app.py", line 279, in run
result = self.run_subcommand(remainder)
File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python3/dist-packages/cliff/display.py", line 116, in run
column_names, data = self.take_action(parsed_args)
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status_cli.py", line 23, in take_action
status = utils.get_client(self).status.get()
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status.py", line 21, in get
return self._get(self.url + '?details=%s' % details).json()
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py", line 37, in _get
return self.client.api.get(*args, **kwargs)
File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 328, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python3/dist-packages/gnocchiclient/client.py", line 52, in request
raise exceptions.from_response(resp, method)
gnocchiclient.exceptions.ClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.17 Port 8041</address>
</body></html>
(HTTP 500)
I have caught a gdb backtrace from the Apache WSGI process after setting threads=1 and processes=1:
Although I have installed the libpython3.6 debug symbols package the frame does not appear to contain useful information.
Including the complete backtrace below:
Thread 4 "(wsgi:gnocchi-a" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fdc337d8700 (LWP 25792)]
0x00007fdc361ba9ee in _PyCFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
(gdb) bt
#0 0x00007fdc361ba9ee in _PyCFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#1 0x00007fdc3620fb61 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#2 0x00007fdc361f50e5 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#3 0x00007fdc28405ffc in __Pyx_PyObject_Call (func=0x7fdc28b31c60, arg=0x7fdbf3424b88, kw=0x7fdbf3406f78)
at ./obj-x86_64-linux-gnu/src/pybind/rados3/pyrex/rados.c:74295
#4 0x00007fdc2848ac14 in __pyx_pf_5rados_8requires_7wrapper_validate_func (__pyx_v_kwargs=0x7fdbf3406f78, __pyx_v_args=0x7fdbf3424b88,
__pyx_self=<optimized out>) at ./obj-x86_64-linux-gnu/src/pybind/rados3/pyrex/rados.c:5406
#5 __pyx_pw_5rados_8requires_7wrapper_1validate_func (__pyx_self=<optimized out>, __pyx_args=0x7fdbf3424b88, __pyx_kwds=<optimized out>)
at ./obj-x86_64-linux-gnu/src/pybind/rados3/pyrex/rados.c:4829
#6 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#7 0x00007fdc36210126 in _PyObject_FastCallKeywords () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#8 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#9 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#10 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#11 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#12 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#13 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#14 0x00007fdc36122eeb in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#15 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#16 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#17 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#18 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#19 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#20 0x00007fdc361230fe in PyEval_EvalCodeEx () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#21 0x00007fdc361e3863 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#22 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#23 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#24 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#25 0x00007fdc36122eeb in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#26 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#27 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#28 0x00007fdc36122eeb in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#29 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#30 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#31 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#32 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#33 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#34 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#35 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#36 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#37 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#38 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#39 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#40 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#41 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#42 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#43 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#44 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#45 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#46 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#47 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
---Type <return> to continue, or q <return> to quit---
#48 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#49 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#50 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#51 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#52 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#53 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#54 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#55 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#56 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#57 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#58 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#59 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#60 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#61 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#62 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#63 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#64 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#65 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#66 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#67 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#68 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#69 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#70 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#71 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#72 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#73 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#74 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#75 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#76 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#77 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#78 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#79 0x00007fdc361228b9 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#80 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#81 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#82 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#83 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#84 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#85 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#86 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#87 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#88 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#89 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#90 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#91 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#92 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#93 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#94 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#95 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#96 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#97 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#98 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
---Type <return> to continue, or q <return> to quit---
#99 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#100 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#101 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#102 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#103 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#104 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#105 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#106 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#107 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#108 0x00007fdc361228b9 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#109 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#110 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#111 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#112 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#113 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#114 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#115 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#116 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#117 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#118 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#119 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#120 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#121 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#122 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#123 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#124 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#125 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#126 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#127 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#128 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#129 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#130 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#131 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#132 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#133 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#134 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#135 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#136 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#137 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#138 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#139 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#140 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#141 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#142 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#143 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#144 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#145 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#146 0x00007fdc361228b9 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#147 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#148 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#149 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
---Type <return> to continue, or q <return> to quit---
#150 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#151 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#152 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#153 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#154 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#155 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#156 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#157 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#158 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#159 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#160 0x00007fdc36671d18 in Adapter_run (object=0x7fdc15bfe3c8, self=<optimized out>) at src/server/mod_wsgi.c:3141
#161 wsgi_execute_script (r=r@entry=0x7fdc3a6940a0) at src/server/mod_wsgi.c:4093
#162 0x00007fdc36674dec in wsgi_hook_daemon_handler (c=<optimized out>) at src/server/mod_wsgi.c:12879
#163 wsgi_process_socket (daemon=0x7fdc3a7c4e60, bucket_alloc=<optimized out>, sock=<optimized out>, p=<optimized out>) at src/server/mod_wsgi.c:8498
#164 wsgi_daemon_worker (thread=0x7fdc3a7e2090, p=<optimized out>) at src/server/mod_wsgi.c:8826
#165 wsgi_daemon_thread (thd=<optimized out>, data=0x7fdc3a7e2090) at src/server/mod_wsgi.c:8913
#166 0x00007fdc39d816db in start_thread (arg=0x7fdc337d8700) at pthread_create.c:463
#167 0x00007fdc39aaa88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) |
[Impact]
gnocchi deployments are broken; API requests fail with a SIGSEGV
[Test Case]
Deploy gnocchi (as part of an openstack cloud)
gnocchi status (500 error code returned)
Alternatively run the functional smoke tests from the gnocchi charm.
[Regression Potential]
Fix for this issue is picked from the upstream VCS and has been released in subsequent point releases for 12.2.x and 13.2.x series of Ceph.
[Original Bug Report]
Running $ gnocchi status --debug
REQ: curl -g -i -X GET http://10.5.0.38:5000/v3 -H "Accept: application/json" -H "User-Agent: gnocchi keystoneauth1/3.10.0 python-requests/2.18.4 CPython/3.6.7"
Starting new HTTP connection (1): 10.5.0.38
http://10.5.0.38:5000 "GET /v3 HTTP/1.1" 200 249
RESP: [200] Connection: Keep-Alive Content-Length: 249 Content-Type: application/json Date: Mon, 28 Jan 2019 11:34:39 GMT Keep-Alive: timeout=5, max=100 Server: Apache/2.4.29 (Ubuntu) Vary: X-Auth-Token X-Distribution: Ubuntu x-openstack-request-id: req-d74445b8-1567-41d0-b741-a27000a80f14
RESP BODY: {"version": {"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "http://10.5.0.38:5000/v3/", "rel": "self"}]}}
GET call to http://10.5.0.38:5000/v3 used request id req-d74445b8-1567-41d0-b741-a27000a80f14
Making authentication request to http://10.5.0.38:5000/v3/auth/tokens
http://10.5.0.38:5000 "POST /v3/auth/tokens HTTP/1.1" 201 1780
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "5a1b23e27bbe47609df2ed5aae809543", "name": "Admin"}, {"id": "9499c375bc694f55891f5c2ed5b89a90", "name": "Member"}], "is_admin_project": true, "project": {"domain": {"id": "d008601304584178a632480f1b5658f1", "name": "admin_domain"}, "id": "efd4af8ba3c147379f140e0fcf03c1a0", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://10.5.0.38:35357/v3", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "564d822b8912406ca8ca86ff0e6fdde3"}, {"url": "http://10.5.0.38:5000/v3", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "56e351a2cff74c24beebf951cea8f099"}, {"url": "http://10.5.0.38:5000/v3", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "f451dcf182b14bdd9955f4d526011bf9"}], "type": "identity", "id": "f4528ee1a55e434488244066eae3bb70", "name": "keystone"}, {"endpoints": [{"url": "http://10.5.0.17:8041", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "93b7789c73f24134971a99cf9773e45c"}, {"url": "http://10.5.0.17:8041", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "e2c3aea790d54df89e4c36b815ce0f18"}, {"url": "http://10.5.0.17:8041", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f20ba6de8db8453bbd57cfc9f0e18e51"}], "type": "metric", "id": "39131ce26e2a40f18b56a89f61d76bce", "name": "gnocchi"}], "expires_at": "2019-01-28T12:34:39.000000Z", "user": {"password_expires_at": null, "domain": {"id": "d008601304584178a632480f1b5658f1", "name": "admin_domain"}, "id": "f656e0c9ff90432ca6a16fbcf689c39b", "name": "admin"}, "audit_ids": ["ORciQMnASuaRQBdBa6uuIg"], "issued_at": "2019-01-28T11:34:39.000000Z"}}
REQ: curl -g -i -X GET http://10.5.0.17:8041/v1/status?details=False -H "Accept: application/json, */*" -H "User-Agent: gnocchi keystoneauth1/3.10.0 python-requests/2.18.4 CPython/3.6.7" -H "X-Auth-Token: {SHA1}f16e953e1742980359603aef5a704611dd23a651"
Starting new HTTP connection (1): 10.5.0.17
http://10.5.0.17:8041 "GET /v1/status?details=False HTTP/1.1" 500 608
RESP: [500] Connection: close Content-Length: 608 Content-Type: text/html; charset=iso-8859-1 Date: Mon, 28 Jan 2019 11:34:39 GMT Server: Apache/2.4.29 (Ubuntu)
RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.17 Port 8041</address>
</body></html>
(HTTP 500)
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python3/dist-packages/cliff/display.py", line 116, in run
column_names, data = self.take_action(parsed_args)
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status_cli.py", line 23, in take_action
status = utils.get_client(self).status.get()
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status.py", line 21, in get
return self._get(self.url + '?details=%s' % details).json()
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py", line 37, in _get
return self.client.api.get(*args, **kwargs)
File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 328, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python3/dist-packages/gnocchiclient/client.py", line 52, in request
raise exceptions.from_response(resp, method)
gnocchiclient.exceptions.ClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.17 Port 8041</address>
</body></html>
(HTTP 500)
Traceback (most recent call last):
File "/usr/bin/gnocchi", line 10, in <module>
sys.exit(main())
File "/usr/lib/python3/dist-packages/gnocchiclient/shell.py", line 251, in main
return GnocchiShell().run(args)
File "/usr/lib/python3/dist-packages/cliff/app.py", line 279, in run
result = self.run_subcommand(remainder)
File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python3/dist-packages/cliff/display.py", line 116, in run
column_names, data = self.take_action(parsed_args)
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status_cli.py", line 23, in take_action
status = utils.get_client(self).status.get()
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status.py", line 21, in get
return self._get(self.url + '?details=%s' % details).json()
File "/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py", line 37, in _get
return self.client.api.get(*args, **kwargs)
File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 328, in get
return self.request(url, 'GET', **kwargs)
File "/usr/lib/python3/dist-packages/gnocchiclient/client.py", line 52, in request
raise exceptions.from_response(resp, method)
gnocchiclient.exceptions.ClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.17 Port 8041</address>
</body></html>
(HTTP 500)
I have caught a gdb backtrace from the Apache WSGI process after setting threads=1 and processes=1:
Although I have installed the libpython3.6 debug symbols package the frame does not appear to contain useful information.
Including the complete backtrace below:
Thread 4 "(wsgi:gnocchi-a" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fdc337d8700 (LWP 25792)]
0x00007fdc361ba9ee in _PyCFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
(gdb) bt
#0 0x00007fdc361ba9ee in _PyCFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#1 0x00007fdc3620fb61 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#2 0x00007fdc361f50e5 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#3 0x00007fdc28405ffc in __Pyx_PyObject_Call (func=0x7fdc28b31c60, arg=0x7fdbf3424b88, kw=0x7fdbf3406f78)
at ./obj-x86_64-linux-gnu/src/pybind/rados3/pyrex/rados.c:74295
#4 0x00007fdc2848ac14 in __pyx_pf_5rados_8requires_7wrapper_validate_func (__pyx_v_kwargs=0x7fdbf3406f78, __pyx_v_args=0x7fdbf3424b88,
__pyx_self=<optimized out>) at ./obj-x86_64-linux-gnu/src/pybind/rados3/pyrex/rados.c:5406
#5 __pyx_pw_5rados_8requires_7wrapper_1validate_func (__pyx_self=<optimized out>, __pyx_args=0x7fdbf3424b88, __pyx_kwds=<optimized out>)
at ./obj-x86_64-linux-gnu/src/pybind/rados3/pyrex/rados.c:4829
#6 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#7 0x00007fdc36210126 in _PyObject_FastCallKeywords () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#8 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#9 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#10 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#11 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#12 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#13 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#14 0x00007fdc36122eeb in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#15 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#16 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#17 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#18 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#19 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#20 0x00007fdc361230fe in PyEval_EvalCodeEx () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#21 0x00007fdc361e3863 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#22 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#23 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#24 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#25 0x00007fdc36122eeb in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#26 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#27 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#28 0x00007fdc36122eeb in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#29 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#30 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#31 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#32 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#33 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#34 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#35 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#36 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#37 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#38 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#39 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#40 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#41 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#42 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#43 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#44 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#45 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#46 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#47 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
---Type <return> to continue, or q <return> to quit---
#48 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#49 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#50 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#51 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#52 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#53 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#54 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#55 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#56 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#57 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#58 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#59 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#60 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#61 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#62 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#63 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#64 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#65 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#66 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#67 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#68 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#69 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#70 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#71 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#72 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#73 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#74 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#75 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#76 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#77 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#78 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#79 0x00007fdc361228b9 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#80 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#81 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#82 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#83 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#84 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#85 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#86 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#87 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#88 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#89 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#90 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#91 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#92 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#93 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#94 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#95 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#96 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#97 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#98 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
---Type <return> to continue, or q <return> to quit---
#99 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#100 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#101 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#102 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#103 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#104 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#105 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#106 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#107 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#108 0x00007fdc361228b9 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#109 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#110 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#111 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#112 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#113 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#114 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#115 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#116 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#117 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#118 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#119 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#120 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#121 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#122 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#123 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#124 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#125 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#126 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#127 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#128 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#129 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#130 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#131 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#132 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#133 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#134 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#135 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#136 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#137 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#138 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#139 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#140 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#141 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#142 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#143 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#144 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#145 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#146 0x00007fdc361228b9 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#147 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#148 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#149 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
---Type <return> to continue, or q <return> to quit---
#150 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#151 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#152 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#153 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#154 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#155 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#156 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#157 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#158 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#159 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#160 0x00007fdc36671d18 in Adapter_run (object=0x7fdc15bfe3c8, self=<optimized out>) at src/server/mod_wsgi.c:3141
#161 wsgi_execute_script (r=r@entry=0x7fdc3a6940a0) at src/server/mod_wsgi.c:4093
#162 0x00007fdc36674dec in wsgi_hook_daemon_handler (c=<optimized out>) at src/server/mod_wsgi.c:12879
#163 wsgi_process_socket (daemon=0x7fdc3a7c4e60, bucket_alloc=<optimized out>, sock=<optimized out>, p=<optimized out>) at src/server/mod_wsgi.c:8498
#164 wsgi_daemon_worker (thread=0x7fdc3a7e2090, p=<optimized out>) at src/server/mod_wsgi.c:8826
#165 wsgi_daemon_thread (thd=<optimized out>, data=0x7fdc3a7e2090) at src/server/mod_wsgi.c:8913
#166 0x00007fdc39d816db in start_thread (arg=0x7fdc337d8700) at pthread_create.c:463
#167 0x00007fdc39aaa88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) |
|
