Update to 12.2.8 leads to SIGSEGV Segmentation fault in python3-rados
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu Cloud Archive |
Fix Released
|
High
|
Unassigned | ||
Queens |
Fix Released
|
High
|
Unassigned | ||
Rocky |
Fix Released
|
High
|
Unassigned | ||
ceph (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Bionic |
Fix Released
|
High
|
James Page | ||
Cosmic |
Fix Released
|
High
|
Unassigned | ||
Disco |
Fix Released
|
High
|
Unassigned |
Bug Description
[Impact]
gnocchi deployments are broken; API requests fail with a SIGSEGV
[Test Case]
Deploy gnocchi (as part of an openstack cloud)
gnocchi status (500 error code returned)
Alternatively run the functional smoke tests from the gnocchi charm.
[Regression Potential]
Fix for this issue is picked from the upstream VCS and has been released in subsequent point releases for 12.2.x and 13.2.x series of Ceph.
[Original Bug Report]
Running $ gnocchi status --debug
REQ: curl -g -i -X GET http://
Starting new HTTP connection (1): 10.5.0.38
http://
RESP: [200] Connection: Keep-Alive Content-Length: 249 Content-Type: application/json Date: Mon, 28 Jan 2019 11:34:39 GMT Keep-Alive: timeout=5, max=100 Server: Apache/2.4.29 (Ubuntu) Vary: X-Auth-Token X-Distribution: Ubuntu x-openstack-
RESP BODY: {"version": {"status": "stable", "updated": "2018-02-
GET call to http://
Making authentication request to http://
http://
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "5a1b23e27bbe47
REQ: curl -g -i -X GET http://
Starting new HTTP connection (1): 10.5.0.17
http://
RESP: [500] Connection: close Content-Length: 608 Content-Type: text/html; charset=iso-8859-1 Date: Mon, 28 Jan 2019 11:34:39 GMT Server: Apache/2.4.29 (Ubuntu)
RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>
</body></html>
(HTTP 500)
Traceback (most recent call last):
File "/usr/lib/
result = cmd.run(
File "/usr/lib/
column_names, data = self.take_
File "/usr/lib/
status = utils.get_
File "/usr/lib/
return self._get(self.url + '?details=%s' % details).json()
File "/usr/lib/
return self.client.
File "/usr/lib/
return self.request(url, 'GET', **kwargs)
File "/usr/lib/
raise exceptions.
gnocchiclient.
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>
</body></html>
(HTTP 500)
Traceback (most recent call last):
File "/usr/bin/gnocchi", line 10, in <module>
sys.
File "/usr/lib/
return GnocchiShell(
File "/usr/lib/
result = self.run_
File "/usr/lib/
result = cmd.run(
File "/usr/lib/
column_names, data = self.take_
File "/usr/lib/
status = utils.get_
File "/usr/lib/
return self._get(self.url + '?details=%s' % details).json()
File "/usr/lib/
return self.client.
File "/usr/lib/
return self.request(url, 'GET', **kwargs)
File "/usr/lib/
raise exceptions.
gnocchiclient.
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>
</body></html>
(HTTP 500)
I have caught a gdb backtrace from the Apache WSGI process after setting threads=1 and processes=1:
Although I have installed the libpython3.6 debug symbols package the frame does not appear to contain useful information.
Including the complete backtrace below:
Thread 4 "(wsgi:gnocchi-a" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fdc337d8700 (LWP 25792)]
0x00007fdc361ba9ee in _PyCFunction_
(gdb) bt
#0 0x00007fdc361ba9ee in _PyCFunction_
#1 0x00007fdc3620fb61 in _PyObject_
#2 0x00007fdc361f50e5 in ?? () from target:
#3 0x00007fdc28405ffc in __Pyx_PyObject_Call (func=0x7fdc28b
at ./obj-x86_
#4 0x00007fdc2848ac14 in __pyx_pf_
__pyx_
#5 __pyx_pw_
at ./obj-x86_
#6 0x00007fdc3620fa99 in _PyObject_
#7 0x00007fdc36210126 in _PyObject_
#8 0x00007fdc36122ae8 in ?? () from target:
#9 0x00007fdc36127e65 in _PyEval_
#10 0x00007fdc3612263f in ?? () from target:
#11 0x00007fdc36122d1e in ?? () from target:
#12 0x00007fdc36127e65 in _PyEval_
#13 0x00007fdc361214a3 in ?? () from target:
#14 0x00007fdc36122eeb in ?? () from target:
#15 0x00007fdc36129092 in _PyEval_
#16 0x00007fdc3612263f in ?? () from target:
#17 0x00007fdc36122d1e in ?? () from target:
#18 0x00007fdc36129092 in _PyEval_
#19 0x00007fdc3612263f in ?? () from target:
#20 0x00007fdc361230fe in PyEval_EvalCodeEx () from target:
#21 0x00007fdc361e3863 in ?? () from target:
#22 0x00007fdc362102d8 in PyObject_Call () from target:
#23 0x00007fdc36127fb5 in _PyEval_
#24 0x00007fdc361214a3 in ?? () from target:
#25 0x00007fdc36122eeb in ?? () from target:
#26 0x00007fdc36129092 in _PyEval_
#27 0x00007fdc361214a3 in ?? () from target:
#28 0x00007fdc36122eeb in ?? () from target:
#29 0x00007fdc36129092 in _PyEval_
#30 0x00007fdc3612263f in ?? () from target:
#31 0x00007fdc36122951 in _PyFunction_
#32 0x00007fdc3620fb41 in _PyObject_
#33 0x00007fdc3621026d in _PyObject_
#34 0x00007fdc362102d8 in PyObject_Call () from target:
#35 0x00007fdc3619beed in ?? () from target:
#36 0x00007fdc3620fa99 in _PyObject_
#37 0x00007fdc36122ae8 in ?? () from target:
#38 0x00007fdc36129092 in _PyEval_
#39 0x00007fdc361214a3 in ?? () from target:
#40 0x00007fdc36122a43 in _PyFunction_
#41 0x00007fdc3620fb41 in _PyObject_
#42 0x00007fdc3621026d in _PyObject_
#43 0x00007fdc362102d8 in PyObject_Call () from target:
#44 0x00007fdc3619beed in ?? () from target:
#45 0x00007fdc3620fa99 in _PyObject_
#46 0x00007fdc36122ae8 in ?? () from target:
#47 0x00007fdc36129092 in _PyEval_
---Type <return> to continue, or q <return> to quit---
#48 0x00007fdc361214a3 in ?? () from target:
#49 0x00007fdc36122a43 in _PyFunction_
#50 0x00007fdc3620fb41 in _PyObject_
#51 0x00007fdc3621026d in _PyObject_
#52 0x00007fdc362102d8 in PyObject_Call () from target:
#53 0x00007fdc3619beed in ?? () from target:
#54 0x00007fdc3620fa99 in _PyObject_
#55 0x00007fdc36122ae8 in ?? () from target:
#56 0x00007fdc36129092 in _PyEval_
#57 0x00007fdc3612263f in ?? () from target:
#58 0x00007fdc36122951 in _PyFunction_
#59 0x00007fdc3620fb41 in _PyObject_
#60 0x00007fdc3621026d in _PyObject_
#61 0x00007fdc362102d8 in PyObject_Call () from target:
#62 0x00007fdc3619beed in ?? () from target:
#63 0x00007fdc3620fa99 in _PyObject_
#64 0x00007fdc36122ae8 in ?? () from target:
#65 0x00007fdc36129092 in _PyEval_
#66 0x00007fdc3612263f in ?? () from target:
#67 0x00007fdc36122d1e in ?? () from target:
#68 0x00007fdc36127e65 in _PyEval_
#69 0x00007fdc3612263f in ?? () from target:
#70 0x00007fdc36122d1e in ?? () from target:
#71 0x00007fdc36129092 in _PyEval_
#72 0x00007fdc361214a3 in ?? () from target:
#73 0x00007fdc36122a43 in _PyFunction_
#74 0x00007fdc3620fb41 in _PyObject_
#75 0x00007fdc3621026d in _PyObject_
#76 0x00007fdc362102d8 in PyObject_Call () from target:
#77 0x00007fdc36127fb5 in _PyEval_
#78 0x00007fdc3612263f in ?? () from target:
#79 0x00007fdc361228b9 in _PyFunction_
#80 0x00007fdc3620fb41 in _PyObject_
#81 0x00007fdc3621026d in _PyObject_
#82 0x00007fdc362102d8 in PyObject_Call () from target:
#83 0x00007fdc36127fb5 in _PyEval_
#84 0x00007fdc3612263f in ?? () from target:
#85 0x00007fdc36122951 in _PyFunction_
#86 0x00007fdc3620fb41 in _PyObject_
#87 0x00007fdc3621026d in _PyObject_
#88 0x00007fdc362102d8 in PyObject_Call () from target:
#89 0x00007fdc3619beed in ?? () from target:
#90 0x00007fdc362102d8 in PyObject_Call () from target:
#91 0x00007fdc3619beed in ?? () from target:
#92 0x00007fdc3620fa99 in _PyObject_
#93 0x00007fdc36122ae8 in ?? () from target:
#94 0x00007fdc36129092 in _PyEval_
#95 0x00007fdc3612263f in ?? () from target:
#96 0x00007fdc36122d1e in ?? () from target:
#97 0x00007fdc36127e65 in _PyEval_
#98 0x00007fdc3612263f in ?? () from target:
---Type <return> to continue, or q <return> to quit---
#99 0x00007fdc36122d1e in ?? () from target:
#100 0x00007fdc36129092 in _PyEval_
#101 0x00007fdc361214a3 in ?? () from target:
#102 0x00007fdc36122a43 in _PyFunction_
#103 0x00007fdc3620fb41 in _PyObject_
#104 0x00007fdc3621026d in _PyObject_
#105 0x00007fdc362102d8 in PyObject_Call () from target:
#106 0x00007fdc36127fb5 in _PyEval_
#107 0x00007fdc3612263f in ?? () from target:
#108 0x00007fdc361228b9 in _PyFunction_
#109 0x00007fdc3620fb41 in _PyObject_
#110 0x00007fdc3621026d in _PyObject_
#111 0x00007fdc362102d8 in PyObject_Call () from target:
#112 0x00007fdc36127fb5 in _PyEval_
#113 0x00007fdc3612263f in ?? () from target:
#114 0x00007fdc36122951 in _PyFunction_
#115 0x00007fdc3620fb41 in _PyObject_
#116 0x00007fdc3621026d in _PyObject_
#117 0x00007fdc362102d8 in PyObject_Call () from target:
#118 0x00007fdc3619beed in ?? () from target:
#119 0x00007fdc362102d8 in PyObject_Call () from target:
#120 0x00007fdc3619beed in ?? () from target:
#121 0x00007fdc3620fa99 in _PyObject_
#122 0x00007fdc36122ae8 in ?? () from target:
#123 0x00007fdc36129092 in _PyEval_
#124 0x00007fdc361214a3 in ?? () from target:
#125 0x00007fdc36122a43 in _PyFunction_
#126 0x00007fdc3620fb41 in _PyObject_
#127 0x00007fdc3621026d in _PyObject_
#128 0x00007fdc362102d8 in PyObject_Call () from target:
#129 0x00007fdc3619beed in ?? () from target:
#130 0x00007fdc3620fa99 in _PyObject_
#131 0x00007fdc36122ae8 in ?? () from target:
#132 0x00007fdc36129092 in _PyEval_
#133 0x00007fdc3612263f in ?? () from target:
#134 0x00007fdc36122d1e in ?? () from target:
#135 0x00007fdc36127e65 in _PyEval_
#136 0x00007fdc3612263f in ?? () from target:
#137 0x00007fdc36122d1e in ?? () from target:
#138 0x00007fdc36129092 in _PyEval_
#139 0x00007fdc361214a3 in ?? () from target:
#140 0x00007fdc36122a43 in _PyFunction_
#141 0x00007fdc3620fb41 in _PyObject_
#142 0x00007fdc3621026d in _PyObject_
#143 0x00007fdc362102d8 in PyObject_Call () from target:
#144 0x00007fdc36127fb5 in _PyEval_
#145 0x00007fdc3612263f in ?? () from target:
#146 0x00007fdc361228b9 in _PyFunction_
#147 0x00007fdc3620fb41 in _PyObject_
#148 0x00007fdc3621026d in _PyObject_
#149 0x00007fdc362102d8 in PyObject_Call () from target:
---Type <return> to continue, or q <return> to quit---
#150 0x00007fdc36127fb5 in _PyEval_
#151 0x00007fdc3612263f in ?? () from target:
#152 0x00007fdc36122951 in _PyFunction_
#153 0x00007fdc3620fb41 in _PyObject_
#154 0x00007fdc3621026d in _PyObject_
#155 0x00007fdc362102d8 in PyObject_Call () from target:
#156 0x00007fdc3619beed in ?? () from target:
#157 0x00007fdc362102d8 in PyObject_Call () from target:
#158 0x00007fdc3619beed in ?? () from target:
#159 0x00007fdc362102d8 in PyObject_Call () from target:
#160 0x00007fdc36671d18 in Adapter_run (object=
#161 wsgi_execute_script (r=r@entry=
#162 0x00007fdc36674dec in wsgi_hook_
#163 wsgi_process_socket (daemon=
#164 wsgi_daemon_worker (thread=
#165 wsgi_daemon_thread (thd=<optimized out>, data=0x7fdc3a7e
#166 0x00007fdc39d816db in start_thread (arg=0x7fdc337d
#167 0x00007fdc39aaa88f in clone () at ../sysdeps/
(gdb)
description: | updated |
Changed in ceph (Ubuntu Cosmic): | |
status: | New → In Progress |
importance: | Undecided → High |
Changed in ceph (Ubuntu Bionic): | |
importance: | Critical → High |
Changed in ceph (Ubuntu Cosmic): | |
assignee: | nobody → James Page (james-page) |
Changed in ceph (Ubuntu Cosmic): | |
assignee: | James Page (james-page) → nobody |
Changed in cloud-archive: | |
status: | New → Fix Released |
importance: | Undecided → High |
Downgrading the ceph packages to 12.2.7 resolves the issue and gnocchi status succeeds