Upgrade to queens results in ImportError: No module named 'rados' nor 'cradox'

[Xav Paice]

After changing openstack-origin from cloud:xenial-pike to cloud:xenial-queens, the logs fill with traceback:

2018-06-28 06:19:21,056 [1306829] ERROR gnocchi.utils: Unable to initialize storage driver
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 298, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/gnocchi/storage/__init__.py", line 117, in get_driver
    conf.storage, coord)
  File "/usr/lib/python3/dist-packages/gnocchi/storage/ceph.py", line 48, in __init__
    self.rados, self.ioctx = ceph.create_rados_connection(conf)
  File "/usr/lib/python3/dist-packages/gnocchi/common/ceph.py", line 51, in create_rados_connection
    raise ImportError("No module named 'rados' nor 'cradox'")
ImportError: No module named 'rados' nor 'cradox'

Package python-rados is installed, but not python3-rados. The gnocchi-metricd process is using Python 3.

If I install python3-rados, the traceback changes:

2018-06-28 06:30:37,027 [1352834] ERROR cotyledon._utils: Unhandled exception
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gnocchi/incoming/__init__.py", line 51, in NUM_SACKS
    self._num_sacks = int(self._get_storage_sacks())
  File "/usr/lib/python3/dist-packages/gnocchi/incoming/ceph.py", line 67, in _get_storage_sacks
    self.ioctx.read(self.CFG_PREFIX).decode())[self.CFG_SACKS]
  File "rados.pyx", line 498, in rados.requires.wrapper.validate_func (/build/ceph-B2ToPL/ceph-12.2.4/obj-x86_64-linux-gnu/src/pybind/rados3/pyrex/rados.c:4922)
  File "rados.pyx", line 2695, in rados.Ioctx.read (/build/ceph-B2ToPL/ceph-12.2.4/obj-x86_64-linux-gnu/src/pybind/rados3/pyrex/rados.c:34873)
rados.ObjectNotFound: [errno 2] Ioctx.read(b''): failed to read b'gnocchi-config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cotyledon/_utils.py", line 95, in exit_on_exception
    yield
  File "/usr/lib/python3/dist-packages/cotyledon/_service.py", line 139, in _run
    self.run()
  File "/usr/lib/python3/dist-packages/gnocchi/cli/metricd.py", line 78, in run
    self._configure()
  File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 214, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 295, in call
    start_time=start_time)
  File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 252, in iter
    return fut.result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 298, in call
    result = fn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/gnocchi/cli/metricd.py", line 163, in _configure
    six.moves.range(self.incoming.NUM_SACKS))
  File "/usr/lib/python3/dist-packages/gnocchi/incoming/__init__.py", line 53, in NUM_SACKS
    raise SackDetectionError(e)
gnocchi.incoming.SackDetectionError: [errno 2] Ioctx.read(b''): failed to read b'gnocchi-config'

[Xav Paice (xavpaice)]

Tried deploying a fresh unit rather than troubleshooting the upgraded ones. Unfortunately, still seeing the second traceback in the logs on a fresh (i.e. not upgraded, started at Queens) unit.

[Xav Paice (xavpaice)]

Running 'gnocchi-upgrade' on one of the units sorted it out. I would have expected the charm to take care of this?

[Xav Paice (xavpaice)]

Also noted on upgraded units, not on recently built ones, which stopped the API from doing anything other than returning a 500:

Thu Jun 28 07:03:39.402967 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] mod_wsgi (pid=1441413): Target WSGI script '/usr/bin/gnocchi-api' cannot be loaded as Python module.
[Thu Jun 28 07:03:39.403022 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] mod_wsgi (pid=1441413): Exception occurred processing WSGI script '/usr/bin/gnocchi-api'.
[Thu Jun 28 07:03:39.403194 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] Traceback (most recent call last):
[Thu Jun 28 07:03:39.403252 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] File "/usr/bin/gnocchi-api", line 22, in <module>
[Thu Jun 28 07:03:39.403257 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] application = app.load_app(api.prepare_service())
[Thu Jun 28 07:03:39.403263 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] File "/usr/lib/python3/dist-packages/gnocchi/rest/app.py", line 173, in load_app
[Thu Jun 28 07:03:39.403266 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] global_conf={'configkey': configkey})
[Thu Jun 28 07:03:39.403271 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] File "/usr/lib/python3/dist-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
[Thu Jun 28 07:03:39.403274 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] return loadobj(APP, uri, name=name, **kw)
[Thu Jun 28 07:03:39.403278 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] File "/usr/lib/python3/dist-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
[Thu Jun 28 07:03:39.403281 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] return context.create()
[Thu Jun 28 07:03:39.403286 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] File "/usr/lib/python3/dist-packages/paste/deploy/loadwsgi.py", line 710, in create
[Thu Jun 28 07:03:39.403289 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] return self.object_type.invoke(self)
[Thu Jun 28 07:03:39.403293 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] File "/usr/lib/python3/dist-packages/paste/deploy/loadwsgi.py", line 144, in invoke
[Thu Jun 28 07:03:39.403305 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] **context.local_conf)
[Thu Jun 28 07:03:39.403310 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] File "/usr/lib/python3/dist-packages/paste/deploy/util.py", line 55, in fix_call
[Thu Jun 28 07:03:39.403313 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] val = callable(*args, **kw)
[Thu Jun 28 07:03:39.403318 2018] [wsgi:error] [pid 1441413:tid 140023696271104] [remote 127.0.0.1:48595] File "/usr/lib/python3/dist-packages/paste/urlmap.py", line 31, in urlmap_factory
[Thu Jun 28 07:03:39....

[James Page (james-page)]

Re your comment in #2 - yes the gnocchi charm should be running the 'gnocchi-upgrade' - so we need to figure out why.

[James Page (james-page)]

I don't think charms.openstack is using the packages list for the target release during the upgrade process:

subprocess.CalledProcessError: Command '['apt-get', '--assume-yes', '--option', 'Dpkg::Options::=--force-confnew', '--option', 'Dpkg::Options::=--force-confdef', 'install', 'gnocchi-api', 'gnocchi-metricd', 'python-apt', 'ceph-common', 'python-rados', 'python-keystonemiddleware', 'apache2', 'libapache2-mod-wsgi', 'memcached', 'python-memcache', 'haproxy']' returned non-zero exit status 100

That should be the packages list for the queens release, but its actually from the prior release class.

[James Page (james-page)]

2018-07-13 11:44:20 DEBUG config-changed Configuration file '/etc/gnocchi/api-paste.ini'
2018-07-13 11:44:20 DEBUG config-changed ==> File on system created by you or by a script.
2018-07-13 11:44:20 DEBUG config-changed ==> File also in package provided by package maintainer.
2018-07-13 11:44:20 DEBUG config-changed ==> Keeping old config file as default.
2018-07-13 11:44:20 DEBUG config-changed
2018-07-13 11:44:20 DEBUG config-changed Configuration file '/etc/gnocchi/policy.json'
2018-07-13 11:44:20 DEBUG config-changed ==> File on system created by you or by a script.
2018-07-13 11:44:20 DEBUG config-changed ==> File also in package provided by package maintainer.
2018-07-13 11:44:20 DEBUG config-changed ==> Keeping old config file as default.

[James Page (james-page)]

With my inflight fixes for target release install of packages:

Commandline: apt-get --assume-yes --option Dpkg::Options::=--force-confnew --option Dpkg::Options::=--force-confdef dist-upgrade

Commandline: apt-get --assume-yes --option Dpkg::Options::=--force-confnew --option Dpkg::Options::=--force-confdef install gnocchi-api gnocchi-metricd python3-apt ceph-common python3-rados python3-keystonemiddleware python3-memcache memcached python-memcache haproxy

[James Page (james-page)]

https://review.openstack.org/582751

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/582751
Committed: https://git.openstack.org/cgit/openstack/charms.openstack/commit/?id=0ac8eb27816bdc59a9e8ed468cab38d1b07b9b33
Submitter: Zuul
Branch: master

commit 0ac8eb27816bdc59a9e8ed468cab38d1b07b9b33
Author: James Page <email address hidden>
Date: Fri Jul 13 12:06:07 2018 +0100

    Ensure target charm is used during upgrades

    When upgrading openstack components, ensure that the charm class
    representing the target release is used for packages and
    config file rendering, supporting changes in config files or
    packages to be installed.

    Also set OPENSTACK_RELEASE_KEY to new release to ensure that
    Charm class resolution in future hook executions observes the
    current release, rather than the release at install.

    Change-Id: I724ae652460e352e24ed7bf2bf57ea181fadff1c
    Closes-Bug: 1779062

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/583345

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/583345
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=bc1745115b6a9c7837075a4353d60a1e5a1e2ca9
Submitter: Zuul
Branch: master

commit bc1745115b6a9c7837075a4353d60a1e5a1e2ca9
Author: James Page <email address hidden>
Date: Tue Jul 17 15:28:40 2018 -0400

    Update WSGI process group

    Ensure WSGI processes run under the 'gnocchi' group to
    support restriction in permissions to /etc/gnocchi/* to
    root or members of the gnocchi group.

    This change also picks up changes in charms.openstack
    to support this functionality, as well as fixing
    misc issues with upgrades.

    Change-Id: I14d865cd96b7c250ad92c8bdee74693a6eccc769
    Closes-Bug: 1780490
    Closes-Bug: 1779062
    Related-Bug: 1781468

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/18.05
Review: https://review.openstack.org/583544

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/583544
Committed: https://git.openstack.org/cgit/openstack/charm-gnocchi/commit/?id=83fd1606e1561c45623b981eaf37f434a9524296
Submitter: Zuul
Branch: stable/18.05

commit 83fd1606e1561c45623b981eaf37f434a9524296
Author: James Page <email address hidden>
Date: Tue Jul 17 15:28:40 2018 -0400

    Update WSGI process group

    Ensure WSGI processes run under the 'gnocchi' group to
    support restriction in permissions to /etc/gnocchi/* to
    root or members of the gnocchi group.

    This change also picks up changes in charms.openstack
    to support this functionality, as well as fixing
    misc issues with upgrades.

    Change-Id: I14d865cd96b7c250ad92c8bdee74693a6eccc769
    Closes-Bug: 1780490
    Closes-Bug: 1779062
    Related-Bug: 1781468
    (cherry picked from commit bc1745115b6a9c7837075a4353d60a1e5a1e2ca9)