Ubuntu
ceph package

Update to 12.2.8 leads to SIGSEGV Segmentation fault in python3-rados

Bionic (18.04)
Bug #1813582

Bug #1813582 reported by Frode Nordahl on 2019-01-28

This bug affects 1 person

	Status	Importance	Assigned to
Ubuntu Cloud Archive	Fix Released	High	Unassigned
Queens	Fix Released	High	Unassigned
Rocky	Fix Released	High	Unassigned
ceph (Ubuntu)	Fix Released	High	Unassigned
Bionic	Fix Released	High	James Page
Cosmic	Fix Released	High	Unassigned
Disco	Fix Released	High	Unassigned

Bug Description

[Impact]
gnocchi deployments are broken; API requests fail with a SIGSEGV

[Test Case]
Deploy gnocchi (as part of an openstack cloud)
gnocchi status (500 error code returned)

Alternatively run the functional smoke tests from the gnocchi charm.

[Regression Potential]
Fix for this issue is picked from the upstream VCS and has been released in subsequent point releases for 12.2.x and 13.2.x series of Ceph.

[Original Bug Report]
Running $ gnocchi status --debug
REQ: curl -g -i -X GET http://10.5.0.38:5000/v3 -H "Accept: application/json" -H "User-Agent: gnocchi keystoneauth1/3.10.0 python-requests/2.18.4 CPython/3.6.7"
Starting new HTTP connection (1): 10.5.0.38
http://10.5.0.38:5000 "GET /v3 HTTP/1.1" 200 249
RESP: [200] Connection: Keep-Alive Content-Length: 249 Content-Type: application/json Date: Mon, 28 Jan 2019 11:34:39 GMT Keep-Alive: timeout=5, max=100 Server: Apache/2.4.29 (Ubuntu) Vary: X-Auth-Token X-Distribution: Ubuntu x-openstack-request-id: req-d74445b8-1567-41d0-b741-a27000a80f14
RESP BODY: {"version": {"status": "stable", "updated": "2018-02-28T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.10", "links": [{"href": "http://10.5.0.38:5000/v3/", "rel": "self"}]}}
GET call to http://10.5.0.38:5000/v3 used request id req-d74445b8-1567-41d0-b741-a27000a80f14
Making authentication request to http://10.5.0.38:5000/v3/auth/tokens
http://10.5.0.38:5000 "POST /v3/auth/tokens HTTP/1.1" 201 1780
{"token": {"is_domain": false, "methods": ["password"], "roles": [{"id": "5a1b23e27bbe47609df2ed5aae809543", "name": "Admin"}, {"id": "9499c375bc694f55891f5c2ed5b89a90", "name": "Member"}], "is_admin_project": true, "project": {"domain": {"id": "d008601304584178a632480f1b5658f1", "name": "admin_domain"}, "id": "efd4af8ba3c147379f140e0fcf03c1a0", "name": "admin"}, "catalog": [{"endpoints": [{"url": "http://10.5.0.38:35357/v3", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "564d822b8912406ca8ca86ff0e6fdde3"}, {"url": "http://10.5.0.38:5000/v3", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "56e351a2cff74c24beebf951cea8f099"}, {"url": "http://10.5.0.38:5000/v3", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "f451dcf182b14bdd9955f4d526011bf9"}], "type": "identity", "id": "f4528ee1a55e434488244066eae3bb70", "name": "keystone"}, {"endpoints": [{"url": "http://10.5.0.17:8041", "interface": "internal", "region": "RegionOne", "region_id": "RegionOne", "id": "93b7789c73f24134971a99cf9773e45c"}, {"url": "http://10.5.0.17:8041", "interface": "admin", "region": "RegionOne", "region_id": "RegionOne", "id": "e2c3aea790d54df89e4c36b815ce0f18"}, {"url": "http://10.5.0.17:8041", "interface": "public", "region": "RegionOne", "region_id": "RegionOne", "id": "f20ba6de8db8453bbd57cfc9f0e18e51"}], "type": "metric", "id": "39131ce26e2a40f18b56a89f61d76bce", "name": "gnocchi"}], "expires_at": "2019-01-28T12:34:39.000000Z", "user": {"password_expires_at": null, "domain": {"id": "d008601304584178a632480f1b5658f1", "name": "admin_domain"}, "id": "f656e0c9ff90432ca6a16fbcf689c39b", "name": "admin"}, "audit_ids": ["ORciQMnASuaRQBdBa6uuIg"], "issued_at": "2019-01-28T11:34:39.000000Z"}}
REQ: curl -g -i -X GET http://10.5.0.17:8041/v1/status?details=False -H "Accept: application/json, */*" -H "User-Agent: gnocchi keystoneauth1/3.10.0 python-requests/2.18.4 CPython/3.6.7" -H "X-Auth-Token: {SHA1}f16e953e1742980359603aef5a704611dd23a651"
Starting new HTTP connection (1): 10.5.0.17
http://10.5.0.17:8041 "GET /v1/status?details=False HTTP/1.1" 500 608
RESP: [500] Connection: close Content-Length: 608 Content-Type: text/html; charset=iso-8859-1 Date: Mon, 28 Jan 2019 11:34:39 GMT Server: Apache/2.4.29 (Ubuntu)
RESP BODY: Omitted, Content-Type is set to text/html; charset=iso-8859-1. Only application/json responses have their bodies logged.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.17 Port 8041</address>
</body></html>
(HTTP 500)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3/dist-packages/cliff/display.py", line 116, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status_cli.py", line 23, in take_action
    status = utils.get_client(self).status.get()
  File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status.py", line 21, in get
    return self._get(self.url + '?details=%s' % details).json()
  File "/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py", line 37, in _get
    return self.client.api.get(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 328, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python3/dist-packages/gnocchiclient/client.py", line 52, in request
    raise exceptions.from_response(resp, method)
gnocchiclient.exceptions.ClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.17 Port 8041</address>
</body></html>
(HTTP 500)
Traceback (most recent call last):
  File "/usr/bin/gnocchi", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/gnocchiclient/shell.py", line 251, in main
    return GnocchiShell().run(args)
  File "/usr/lib/python3/dist-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python3/dist-packages/cliff/display.py", line 116, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status_cli.py", line 23, in take_action
    status = utils.get_client(self).status.get()
  File "/usr/lib/python3/dist-packages/gnocchiclient/v1/status.py", line 21, in get
    return self._get(self.url + '?details=%s' % details).json()
  File "/usr/lib/python3/dist-packages/gnocchiclient/v1/base.py", line 37, in _get
    return self.client.api.get(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 328, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python3/dist-packages/gnocchiclient/client.py", line 52, in request
    raise exceptions.from_response(resp, method)
gnocchiclient.exceptions.ClientException: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
[no address given] to inform them of the time this error occurred,
and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at 10.5.0.17 Port 8041</address>
</body></html>
(HTTP 500)

I have caught a gdb backtrace from the Apache WSGI process after setting threads=1 and processes=1:

Although I have installed the libpython3.6 debug symbols package the frame does not appear to contain useful information.

Including the complete backtrace below:
Thread 4 "(wsgi:gnocchi-a" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fdc337d8700 (LWP 25792)]
0x00007fdc361ba9ee in _PyCFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
(gdb) bt
#0 0x00007fdc361ba9ee in _PyCFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#1 0x00007fdc3620fb61 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#2 0x00007fdc361f50e5 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#3 0x00007fdc28405ffc in __Pyx_PyObject_Call (func=0x7fdc28b31c60, arg=0x7fdbf3424b88, kw=0x7fdbf3406f78)
    at ./obj-x86_64-linux-gnu/src/pybind/rados3/pyrex/rados.c:74295
#4 0x00007fdc2848ac14 in __pyx_pf_5rados_8requires_7wrapper_validate_func (__pyx_v_kwargs=0x7fdbf3406f78, __pyx_v_args=0x7fdbf3424b88,
    __pyx_self=<optimized out>) at ./obj-x86_64-linux-gnu/src/pybind/rados3/pyrex/rados.c:5406
#5 __pyx_pw_5rados_8requires_7wrapper_1validate_func (__pyx_self=<optimized out>, __pyx_args=0x7fdbf3424b88, __pyx_kwds=<optimized out>)
    at ./obj-x86_64-linux-gnu/src/pybind/rados3/pyrex/rados.c:4829
#6 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#7 0x00007fdc36210126 in _PyObject_FastCallKeywords () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#8 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#9 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#10 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#11 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#12 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#13 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#14 0x00007fdc36122eeb in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#15 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#16 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#17 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#18 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#19 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#20 0x00007fdc361230fe in PyEval_EvalCodeEx () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#21 0x00007fdc361e3863 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#22 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#23 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#24 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#25 0x00007fdc36122eeb in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#26 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#27 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#28 0x00007fdc36122eeb in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#29 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#30 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#31 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#32 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#33 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#34 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#35 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#36 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#37 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#38 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#39 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#40 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#41 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#42 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#43 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#44 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#45 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#46 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#47 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
---Type <return> to continue, or q <return> to quit---
#48 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#49 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#50 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#51 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#52 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#53 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#54 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#55 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#56 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#57 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#58 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#59 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#60 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#61 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#62 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#63 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#64 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#65 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#66 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#67 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#68 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#69 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#70 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#71 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#72 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#73 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#74 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#75 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#76 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#77 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#78 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#79 0x00007fdc361228b9 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#80 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#81 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#82 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#83 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#84 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#85 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#86 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#87 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#88 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#89 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#90 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#91 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#92 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#93 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#94 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#95 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#96 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#97 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#98 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
---Type <return> to continue, or q <return> to quit---
#99 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#100 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#101 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#102 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#103 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#104 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#105 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#106 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#107 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#108 0x00007fdc361228b9 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#109 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#110 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#111 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#112 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#113 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#114 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#115 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#116 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#117 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#118 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#119 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#120 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#121 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#122 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#123 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#124 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#125 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#126 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#127 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#128 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#129 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#130 0x00007fdc3620fa99 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#131 0x00007fdc36122ae8 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#132 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#133 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#134 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#135 0x00007fdc36127e65 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#136 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#137 0x00007fdc36122d1e in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#138 0x00007fdc36129092 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#139 0x00007fdc361214a3 in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#140 0x00007fdc36122a43 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#141 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#142 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#143 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#144 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#145 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#146 0x00007fdc361228b9 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#147 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#148 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#149 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
---Type <return> to continue, or q <return> to quit---
#150 0x00007fdc36127fb5 in _PyEval_EvalFrameDefault () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#151 0x00007fdc3612263f in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#152 0x00007fdc36122951 in _PyFunction_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#153 0x00007fdc3620fb41 in _PyObject_FastCallDict () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#154 0x00007fdc3621026d in _PyObject_Call_Prepend () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#155 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#156 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#157 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#158 0x00007fdc3619beed in ?? () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#159 0x00007fdc362102d8 in PyObject_Call () from target:/usr/lib/x86_64-linux-gnu/libpython3.6m.so.1.0
#160 0x00007fdc36671d18 in Adapter_run (object=0x7fdc15bfe3c8, self=<optimized out>) at src/server/mod_wsgi.c:3141
#161 wsgi_execute_script (r=r@entry=0x7fdc3a6940a0) at src/server/mod_wsgi.c:4093
#162 0x00007fdc36674dec in wsgi_hook_daemon_handler (c=<optimized out>) at src/server/mod_wsgi.c:12879
#163 wsgi_process_socket (daemon=0x7fdc3a7c4e60, bucket_alloc=<optimized out>, sock=<optimized out>, p=<optimized out>) at src/server/mod_wsgi.c:8498
#164 wsgi_daemon_worker (thread=0x7fdc3a7e2090, p=<optimized out>) at src/server/mod_wsgi.c:8826
#165 wsgi_daemon_thread (thd=<optimized out>, data=0x7fdc3a7e2090) at src/server/mod_wsgi.c:8913
#166 0x00007fdc39d816db in start_thread (arg=0x7fdc337d8700) at pthread_create.c:463
#167 0x00007fdc39aaa88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

See original description

Tags:

Revision history for this message

Frode Nordahl (fnordahl) wrote on 2019-01-28:

Downgrading the ceph packages to 12.2.7 resolves the issue and gnocchi status succeeds

Revision history for this message

James Page (james-page) wrote on 2019-01-28:

Maybe http://tracker.ceph.com/issues/25175

Revision history for this message

James Page (james-page) wrote on 2019-01-28:

I think this is fixed in 12.2.9 - we should probably just rev up to the latest 12.2.x release to resolve.

Revision history for this message

James Page (james-page) wrote on 2019-01-28:

Test package for bionic building here:

https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3534

Changed in ceph (Ubuntu):
status:	New → In Progress
importance:	Undecided → Critical
assignee:	nobody → James Page (james-page)
Changed in ceph (Ubuntu Bionic):
status:	New → In Progress
importance:	Undecided → Critical
assignee:	nobody → James Page (james-page)
Changed in ceph (Ubuntu):
status:	In Progress → Invalid
importance:	Critical → Undecided
assignee:	James Page (james-page) → nobody
Changed in ceph (Ubuntu Disco):
status:	Invalid → New

Revision history for this message

James Page (james-page) wrote on 2019-01-28:

May impact cosmic and disco - checking now.

Changed in ceph (Ubuntu Disco):
status:	New → Fix Released

Revision history for this message

James Page (james-page) wrote on 2019-01-28:

disco has the latest mimic release which contains the proposed fix for this issue so should be fine - marking Fix Released.

Revision history for this message

James Page (james-page) wrote on 2019-01-28:

cosmic is still on 13.2.1 so will potentially be impacted.

Revision history for this message

Frode Nordahl (fnordahl) wrote on 2019-01-29:

Download full text (10.3 KiB)

I can confirm that using the packages from PPA mentioned in #4 solves the issue for Bionic:

$ tox -c build/builds/gnocchi/tox.ini -e func27-smoke --workdir /tmp/tox
func27-smoke installed: DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.,amulet==1.21.0,aodhclient==1.1.1,appdirs==1.4.3,asn1crypto==0.24.0,Babel==2.6.0,backports.os==0.1.1,blessings==1.6,bundletester==0.12.2,certifi==2018.11.29,cffi==1.11.5,chardet==3.0.4,charm-tools==2.4.5,charmhelpers==0.19.6,Cheetah3==3.1.0,cliff==2.14.0,cmd2==0.8.9,colander==1.0b1,configparser==3.7.1,contextlib2==0.5.5,coverage==4.5.2,cryptography==2.5,debtcollector==1.20.0,decorator==4.3.2,distro==1.3.0,distro-info==0.0.0,docutils==0.14,dogpile.cache==0.7.1,entrypoints==0.3,enum34==1.1.6,extras==1.0.0,fixtures==3.0.0,flake8==2.4.1,funcsigs==1.0.2,functools32==3.2.3.post2,future==0.17.1,futures==3.2.0,futurist==1.8.0,gnocchiclient==3.1.1,httplib2==0.12.0,idna==2.8,importlib-metadata==0.8,ipaddress==1.0.22,iso8601==0.1.12,Jinja2==2.10,jmespath==0.9.3,jsonpatch==1.23,jsonpointer==2.0,jsonschema==2.5.1,juju-deployer==0.11.0,juju-wait==2.5.0,jujubundlelib==0.5.6,jujuclient==0.54.0,keyring==17.1.1,keystoneauth1==3.11.2,launchpadlib==1.10.6,lazr.authentication==0.1.3,lazr.restfulclient==0.14.2,lazr.uri==1.0.3,libcharmstore==0.0.9,linecache2==1.0.0,macaroonbakery==1.2.1,MarkupSafe==1.1.0,mccabe==0.3.1,mock==2.0.0,monotonic==1.5,msgpack==0.6.1,munch==2.3.2,netaddr==0.7.19,netifaces==0.10.9,nose==1.3.7,oauth==1.0.1,oauthlib==3.0.1,openstacksdk==0.23.0,os-client-config==1.31.2,os-service-types==1.5.0,os-testr==1.0.0,osc-lib==1.12.0,oslo.config==6.8.0,oslo.context==2.22.0,oslo.i18n==3.23.0,oslo.log==3.42.2,oslo.serialization==2.28.1,oslo.utils==3.40.1,otherstuf==1.1.0,parse==1.11.1,path.py==11.5.0,pathlib2==2.3.3,pathspec==0.3.4,pbr==5.1.1,pep8==1.7.1,pika==0.13.0,pkg-resources==0.0.0,prettytable==0.7.2,protobuf==3.6.1,pycparser==2.19,pyflakes==0.8.1,pyinotify==0.9.6,pymacaroons==0.13.0,PyNaCl==1.3.0,pyOpenSSL==19.0.0,pyparsing==2.3.1,pyperclip==1.7.0,pyRFC3339==1.1,python-barbicanclient==4.8.1,python-ceilometerclient==2.9.0,python-cinderclient==4.1.0,python-dateutil==2.7.5,python-designateclient==2.11.0,python-glanceclient==2.15.0,python-heatclient==1.17.0,python-keystoneclient==3.18.0,python-manilaclient==1.26.0,python-mimeparse==1.6.0,python-neutronclient==6.11.0,python-novaclient==11.1.0,python-openstackclient==3.17.0,python-subunit==1.3.0,python-swiftclient==3.6.0,pytz==2018.9,pyudev==0.21.0,PyYAML==3.11,requests==2.21.0,requestsexceptions==1.4.0,rfc3986==1.2.0,ruamel.base==1.0.0,ruamel.ordereddict==0.4.13,ruamel.yaml==0.10.23,scandir==1.9.0,SecretStorage==2.3.1,simplejson==3.16.0,six==1.12.0,stestr==2.2.0,stevedore==1.30.0,stuf==0.9.16,subprocess32==3.5.3,Tempita==0.5.2,testresources==2.0.1,testtools==2.3.0,theblues==0.5.1,traceback2==1.4.0,translationstring==1.3,unicodecsv==0.14.1,unittest2==1.1.0,urllib3==1.24.1,virtualenv==16.3.0,voluptuous==0.11.5,wadllib==1.3.3,warlock==1.3.0,wcwidth==0.1.7,websocket-client==0.40.0,wrapt==1.11.1,wsgi-intercept==1.8.0,zipp==0.3.3,zope.interface==4.6.0
func27-smoke runtests: PYTHONHASHSEED='0'
func27-smoke runtests: commands[0] | bundletester -vl DEBUG -r json -o func-results.json gate-basic-bionic-queens --no-destroy
DEBUG:bundletester.utils:Updating JUJU_MODEL: "" -> "fnordahl-serverstack:admin/default"
DEBUG:root:Bootstrap environment: fnordahl-serverstack:admin/default
DEBUG:deployer.env:Connecting to fnordahl-serverstack:admin/default...
DEBUG:jujuclient.connector:Connecting to wss://10.5.0.3:17070/model/8bab6f50-03a0-4ef8-8df1-1bf865dfae3f/api
DEBUG:deployer.env:Connected.
DEBUG:deployer.env: Terminating machines forcefully
INFO:deployer.env:  Waiting for machine termination
DEBUG:jujuclient.connector:Connecting to wss://10.5.0.3:17070/model/8bab6f50-03a0-4ef8-8df1-1bf865dfae3f/api
DEBUG:root:Waiting for applications to be removed...
DEBUG:runner:call ['/tmp/tox/func27-smoke/bin/charm-proof'] (cwd: /tmp/bundletester-_FRmpX/gnocchi)
DEBUG:runner:I: `display-name` not provided, add for custom naming in the UI
DEBUG:runner:I: config.yaml: option ssl_key has no default value
DEBUG:runner:I: config.yaml: option os-admin-network has no default value
DEBUG:runner:I: config.yaml: option haproxy-server-timeout has no default value
DEBUG:runner:I: config.yaml: option os-public-hostname has no default value
DEBUG:runner:I: config.yaml: option vip has no default value
DEBUG:runner:I: config.yaml: option os-admin-hostname has no default value
DEBUG:runner:I: config.yaml: option haproxy-client-timeout has no default value
DEBUG:runner:I: config.yaml: option ssl_ca has no default value
DEBUG:runner:I: config.yaml: option ssl_cert has no default value
DEBUG:runner:I: config.yaml: option os-internal-network has no default value
DEBUG:runner:I: config.yaml: option haproxy-queue-timeout has no default value
DEBUG:runner:I: config.yaml: option worker-multiplier has no default value
DEBUG:runner:I: config.yaml: option os-public-network has no default value
DEBUG:runner:I: config.yaml: option os-internal-hostname has no default value
DEBUG:runner:I: config.yaml: option haproxy-connect-timeout has no default value
DEBUG:runner:Exit Code: 0
DEBUG:deployer.env: Terminating machines forcefully
INFO:deployer.env:  Waiting for machine termination
DEBUG:jujuclient.connector:Connecting to wss://10.5.0.3:17070/model/8bab6f50-03a0-4ef8-8df1-1bf865dfae3f/api
DEBUG:root:Waiting for applications to be removed...
DEBUG:runner:call ['/tmp/bundletester-_FRmpX/gnocchi/tests/gate-basic-bionic-queens'] (cwd: /tmp/bundletester-_FRmpX/gnocchi)
DEBUG:runner:2019-01-29 07:18:51,301 __init__ INFO: OpenStackAmuletDeployment:  init
DEBUG:runner:2019-01-29 07:18:51,301 _add_services INFO: OpenStackAmuletDeployment:  adding services
DEBUG:runner:2019-01-29 07:18:51,301 _determine_branch_locations INFO: OpenStackAmuletDeployment:  determine branch locations
DEBUG:runner:2019-01-29 07:18:53,545 _configure_services INFO: OpenStackAmuletDeployment:  configure services
DEBUG:runner:2019-01-29 07:18:54 Starting deployment of fnordahl-serverstack:admin/default
DEBUG:runner:2019-01-29 07:18:54 Deploying applications...
DEBUG:runner:2019-01-29 07:18:54  Deploying application ceilometer using cs:~openstack-charmers-next/ceilometer-323
DEBUG:runner:2019-01-29 07:18:57  Deploying application ceph-mon using cs:~openstack-charmers-next/ceph-mon-362
DEBUG:runner:2019-01-29 07:18:59  Deploying application ceph-osd using cs:~openstack-charmers-next/ceph-osd-385
DEBUG:runner:2019-01-29 07:19:02  Deploying application gnocchi using /tmp/charmrUslbY/bionic/gnocchi
DEBUG:runner:2019-01-29 07:19:05  Deploying application keystone using cs:~openstack-charmers-next/keystone-405
DEBUG:runner:2019-01-29 07:19:08  Deploying application memcached using cs:memcached-23
DEBUG:runner:2019-01-29 07:19:10  Deploying application percona-cluster using cs:~openstack-charmers-next/percona-cluster-325
DEBUG:runner:2019-01-29 07:19:12  Deploying application rabbitmq-server using cs:~openstack-charmers-next/rabbitmq-server-331
DEBUG:runner:2019-01-29 07:24:59 Adding relations...
DEBUG:runner:2019-01-29 07:24:59  Adding relation gnocchi:shared-db <-> percona-cluster:shared-db
DEBUG:runner:2019-01-29 07:24:59  Adding relation ceilometer:identity-credentials <-> keystone:identity-credentials
DEBUG:runner:2019-01-29 07:24:59  Adding relation ceph-mon:osd <-> ceph-osd:mon
DEBUG:runner:2019-01-29 07:24:59  Adding relation keystone:shared-db <-> percona-cluster:shared-db
DEBUG:runner:2019-01-29 07:24:59  Adding relation gnocchi:metric-service <-> ceilometer:metric-service
DEBUG:runner:2019-01-29 07:24:59  Adding relation gnocchi:identity-service <-> keystone:identity-service
DEBUG:runner:2019-01-29 07:24:59  Adding relation gnocchi:storage-ceph <-> ceph-mon:client
DEBUG:runner:2019-01-29 07:24:59  Adding relation gnocchi:coordinator-memcached <-> memcached:cache
DEBUG:runner:2019-01-29 07:25:00  Adding relation ceilometer:amqp <-> rabbitmq-server:amqp
DEBUG:runner:2019-01-29 07:28:10 Deployment complete in 557.11 seconds
DEBUG:runner:2019-01-29 07:29:53,536 __init__ INFO: Waiting on extended status checks...
DEBUG:runner:2019-01-29 07:29:53,537 _auto_wait_for_status INFO: Waiting for extended status on units for 5400s...
DEBUG:runner:2019-01-29 07:29:53,538 _auto_wait_for_status DEBUG: Default extended status wait match:  contains READY (case-insensitive)
DEBUG:runner:2019-01-29 07:29:53,538 _auto_wait_for_status DEBUG: Excluding services from extended status match: ['mysql', 'mongodb', 'memcached']
DEBUG:runner:2019-01-29 07:29:53,538 _auto_wait_for_status DEBUG: Waiting up to 5400s for extended status on services: ['ceilometer', 'percona-cluster', 'rabbitmq-server', 'keystone', 'ceph-osd', 'ceph-mon', 'gnocchi']
DEBUG:runner:2019-01-29 07:29:55,643 _auto_wait_for_status INFO: OK
DEBUG:runner:2019-01-29 07:29:55,644 get_default_keystone_session DEBUG: Authenticating keystone admin...
DEBUG:runner:2019-01-29 07:29:56,534 test_100_services DEBUG: Checking system services on units...
DEBUG:runner:2019-01-29 07:29:56,535 validate_services_by_name DEBUG: Checking status of system services...
DEBUG:runner:2019-01-29 07:29:58,752 get_ubuntu_release_from_sentry DEBUG: gnocchi/0 lsb_release: bionic
DEBUG:runner:2019-01-29 07:29:59,974 validate_services_by_name DEBUG: gnocchi/0 `sudo service haproxy status` returned 0
DEBUG:runner:2019-01-29 07:30:01,208 validate_services_by_name DEBUG: gnocchi/0 `sudo service gnocchi-metricd status` returned 0
DEBUG:runner:2019-01-29 07:30:02,446 validate_services_by_name DEBUG: gnocchi/0 `sudo service apache2 status` returned 0
DEBUG:runner:2019-01-29 07:30:02,446 test_100_services DEBUG: OK
DEBUG:runner:2019-01-29 07:30:02,447 test_200_api_connection DEBUG: Checking api functionality...
DEBUG:runner:2019-01-29 07:30:04,962 test_200_api_connection DEBUG: OK
DEBUG:runner:2019-01-29 07:30:08,594 run_action WARNING: charmhelpers.contrib.amulet.utils.run_action has been deprecated for amulet.run_action
DEBUG:runner:2019-01-29 07:30:20,643 run_action WARNING: charmhelpers.contrib.amulet.utils.run_action has been deprecated for amulet.run_action
DEBUG:runner:Exit Code: 0
DEBUG:bundletester.utils:Updating JUJU_MODEL: "fnordahl-serverstack:admin/default" -> ""
_________________________________________________________ summary __________________________________________________________
  func27-smoke: commands succeeded
  congratulations :)

James Page (james-page) on 2019-01-29

description:	updated
Changed in ceph (Ubuntu Cosmic):
status:	New → In Progress
importance:	Undecided → High
Changed in ceph (Ubuntu Bionic):
importance:	Critical → High
Changed in ceph (Ubuntu Cosmic):
assignee:	nobody → James Page (james-page)

Revision history for this message

James Page (james-page) wrote on 2019-01-29:

Bionic update uploaded for SRU team review; I already had a point release in flight for cosmic so will upload that referencing this bug shortly.

Revision history for this message

James Page (james-page) wrote on 2019-01-29:

#10

Cosmic is not actually impacted by this bug so marking as Invalid for >= Cosmic

Changed in ceph (Ubuntu Cosmic):
status:	In Progress → Invalid
Changed in ceph (Ubuntu Disco):
status:	Fix Released → Invalid
Changed in ceph (Ubuntu Cosmic):
status:	Invalid → Fix Released
Changed in ceph (Ubuntu Disco):
status:	Invalid → Fix Released
importance:	Undecided → High

Revision history for this message

James Page (james-page) wrote on 2019-01-29:

#11

Actually make that Fix Released rather than invalid - changes in gnocchi or ceph have resolved this issue in later versions.

James Page (james-page) on 2019-01-29

Changed in ceph (Ubuntu Cosmic):
assignee:	James Page (james-page) → nobody

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2019-01-31: Please test proposed package

#12

Hello Frode, or anyone else affected,

Accepted ceph into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceph/12.2.8-0ubuntu0.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in ceph (Ubuntu Bionic):
status:	In Progress → Fix Committed
tags:	added: verification-needed verification-needed-bionic

Revision history for this message

Frode Nordahl (fnordahl) wrote on 2019-01-31:

#13

Download full text (4.3 KiB)

DEBUG:runner:2019-01-31 14:44:29 Deployment complete in 606.56 seconds
DEBUG:runner:2019-01-31 14:45:34,400 __init__ INFO: Waiting on extended status checks...
DEBUG:runner:2019-01-31 14:45:34,402 _auto_wait_for_status INFO: Waiting for extended status on units for 5400s...
DEBUG:runner:2019-01-31 14:45:34,402 _auto_wait_for_status DEBUG: Default extended status wait match: contains READY (case-insensitive)
DEBUG:runner:2019-01-31 14:45:34,402 _auto_wait_for_status DEBUG: Excluding services from extended status match: ['mysql', 'mongodb', 'memcached', 'ceilometer']
DEBUG:runner:2019-01-31 14:45:34,403 _auto_wait_for_status DEBUG: Waiting up to 5400s for extended status on services: ['percona-cluster', 'rabbitmq-server', 'keystone', 'ceph-osd', 'ceph-mon', 'gnocchi']
DEBUG:runner:2019-01-31 14:45:37,063 _auto_wait_for_status INFO: OK
DEBUG:runner:2019-01-31 14:45:37,064 get_default_keystone_session DEBUG: Authenticating keystone admin...
DEBUG:runner:2019-01-31 14:45:37,876 run_ceilometer_upgrade_action DEBUG: Checking ceilometer-upgrade
DEBUG:runner:2019-01-31 14:46:06,343 _auto_wait_for_status INFO: Waiting for extended status on units for 5400s...
DEBUG:runner:2019-01-31 14:46:06,344 _auto_wait_for_status DEBUG: Default extended status wait match: contains READY (case-insensitive)
DEBUG:runner:2019-01-31 14:46:06,344 _auto_wait_for_status DEBUG: Excluding services from extended status match: ['mysql', 'mongodb', 'memcached']
DEBUG:runner:2019-01-31 14:46:06,344 _auto_wait_for_status DEBUG: Waiting up to 5400s for extended status on services: ['ceilometer', 'percona-cluster', 'rabbitmq-server', 'keystone', 'ceph-osd', 'ceph-mon', 'gnocchi']
DEBUG:runner:2019-01-31 14:46:37,214 _auto_wait_for_status INFO: OK
DEBUG:runner:2019-01-31 14:46:37,214 run_ceilometer_upgrade_action DEBUG: OK
DEBUG:runner:2019-01-31 14:46:37,214 test_100_services DEBUG: Checking system services on units...
DEBUG:runner:2019-01-31 14:46:37,214 validate_services_by_name DEBUG: Checking status of system services...
DEBUG:runner:2019-01-31 14:46:41,478 get_ubuntu_release_from_sentry DEBUG: gnocchi/0 lsb_release: bionic
DEBUG:runner:2019-01-31 14:46:42,763 validate_services_by_name DEBUG: gnocchi/0 `sudo service haproxy status` returned 0
DEBUG:runner:2019-01-31 14:46:44,045 validate_services_by_name DEBUG: gnocchi/0 `sudo service gnocchi-metricd status` returned 0
DEBUG:runner:2019-01-31 14:46:45,348 validate_services_by_name DEBUG: gnocchi/0 `sudo service apache2 status` returned 0
DEBUG:runner:2019-01-31 14:46:45,349 test_100_services DEBUG: OK
DEBUG:runner:2019-01-31 14:46:45,349 test_200_api_connection DEBUG: Checking api functionality...
DEBUG:runner:2019-01-31 14:46:45,960 test_200_api_connection DEBUG: OK
DEBUG:runner:2019-01-31 14:46:49,735 run_action WARNING: charmhelpers.contrib.amulet.utils.run_action has been deprecated for amulet.run_action
DEBUG:runner:2019-01-31 14:47:01,997 run_action WARNING: charmhelpers.contrib.amulet.utils.run_action has been deprecated for amulet.run_action
DEBUG:runner:Exit Code: 0
DEBUG:bundletester.utils:Updating JUJU_MODEL: "fnordahl-serverstack:admin/default" -> ""
_______________________________________________________________...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Changed in ceph (Ubuntu Cosmic):
status:	Fix Released → New

Changed in cloud-archive:
status:	New → Fix Released
importance:	Undecided → High

Changed in ceph (Ubuntu Bionic):
status:	Fix Committed → Fix Released

Ubuntuceph package

Update to 12.2.8 leads to SIGSEGV Segmentation fault in python3-rados

Bug Description

Other bug subscribers

Remote bug watches

Ubuntu
ceph package