gnocchi-api missing rados and ceph dependencies

[James Shelby]

When enabling gnocchi in kolla-ansible I am running in to this error on the glance-api container.

2021-05-07 15:41:49,733 [21] ERROR gnocchi.utils: Unable to initialize storage driver
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/tenacity/__init__.py", line 426, in __call__
    result = fn(*args, **kwargs)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/gnocchi/storage/__init__.py", line 102, in get_driver
    conf.storage)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/gnocchi/storage/ceph.py", line 52, in __init__
    self.rados, self.ioctx = ceph.create_rados_connection(conf)
  File "/var/lib/kolla/venv/lib/python3.6/site-packages/gnocchi/common/ceph.py", line 46, in create_rados_connection
    raise ImportError("No module named 'rados'")
ImportError: No module named 'rados'

This appears that the python3-rados package is missing and possibly a few other dependencies.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla/+/791122

[Mark Goddard (mgoddard)]

Hi James, can you provide more info about your setup? Which release are you using? Which distribution? source/binary images?

At least in binary images on master we have python3-rados. It does look like source images do not though.

I proposed https://review.opendev.org/c/openstack/kolla/+/791122 as a potential fix.

[Boris Lukashev (rageltman)]

@mgoddard: This happens on the current wallaby branch when telling gnocchi to use the ceph backend:
```
2021-06-30 13:43:50,974 [18] ERROR gnocchi.utils: Unable to initialize storage driver
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __call__
    result = fn(*args, **kwargs)
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/gnocchi/storage/__init__.py", line 101, in get_driver
    return utils.get_driver_class('gnocchi.storage', conf.storage)(
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/gnocchi/storage/ceph.py", line 52, in __init__
    self.rados, self.ioctx = ceph.create_rados_connection(conf)
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/gnocchi/common/ceph.py", line 46, in create_rados_connection
    raise ImportError("No module named 'rados'")
ImportError: No module named 'rados'
```
Whats really fun is that the docker process doesnt fail, just loops like this, forever.

How does one hot-patch a deployment with the patch under review? Seems the relevant Dockerfile template doesn't exist in the venv on the machine from which ansible is run - still a bit new to the process, coming over the from the Juju paradigm (had to drop it after internal snap errors killed etcd which killed vault which killed mysql and ceph) for the last stack and generally a Chef person when it comes to devops so between the kolla and ansible pieces, i'm still working my way through structure.

[Boris Lukashev (rageltman)]

To work around this, i tried to use:
```
diff --git a/ansible/roles/gnocchi/defaults/main.yml b/ansible/roles/gnocchi/defaults/main.yml
index 7ebfb079a..fea359beb 100644
--- a/ansible/roles/gnocchi/defaults/main.yml
+++ b/ansible/roles/gnocchi/defaults/main.yml
@@ -57,7 +57,7 @@ gnocchi_database_address: "{{ database_address | put_address_in_context('url') }
 ####################
 # Docker
 ####################
-gnocchi_install_type: "{{ kolla_install_type }}"
+gnocchi_install_type: "binary"
 gnocchi_tag: "{{ openstack_tag }}"

 gnocchi_api_image: "{{ docker_registry ~ '/' if docker_registry else '' }}{{ docker_namespace }}/{{ kolla_base_distro }}-{{ gnocchi_install_type }}-gnocchi-api"
```
in order to use just this container as binary, but no dice, goes back into the missing rados library loop, so clearly still trying to build the container from scratch.

Is this the wrong approach to try and get a single image from upstream?

[Mark Goddard (mgoddard)]

@Boris, I created a patch that installs python3-rados in the image. I wasn't able to get gnocchi working in CI however, so the patch hasn't merged.

If you can test by manually installing the package, that will be sufficient to get the patch merged.

CentOS:

sudo docker exec -u root -it gnocchi_api dnf -y install python3-rados

Debian/Ubuntu:

sudo docker exec -u root -it gnocchi_api apt update
sudo docker exec -u root -it gnocchi_api apt -y install python3-rados

[Boris Lukashev (rageltman)]

Thank you sir.

There is no gnocchi_api container, only a gnocchi_bootstrap one.
I adapted the ubuntu invocation you provided to the container name, it completed the update, and installed python3-rados in the space between these two log outputs:
```
2021-07-01 23:34:44,892 [21] ERROR gnocchi.utils: Unable to initialize storage driver
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __call__
    result = fn(*args, **kwargs)
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/gnocchi/storage/__init__.py", line 101, in get_driver
    return utils.get_driver_class('gnocchi.storage', conf.storage)(
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/gnocchi/storage/ceph.py", line 52, in __init__
    self.rados, self.ioctx = ceph.create_rados_connection(conf)
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/gnocchi/common/ceph.py", line 46, in create_rados_connection
    raise ImportError("No module named 'rados'")
ImportError: No module named 'rados'

2021-07-01 23:35:44,955 [21] ERROR gnocchi.utils: Unable to initialize storage driver
Traceback (most recent call last):
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/tenacity/__init__.py", line 426, in __call__
    result = fn(*args, **kwargs)
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/gnocchi/storage/__init__.py", line 101, in get_driver
    return utils.get_driver_class('gnocchi.storage', conf.storage)(
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/gnocchi/storage/ceph.py", line 52, in __init__
    self.rados, self.ioctx = ceph.create_rados_connection(conf)
  File "/var/lib/kolla/venv/lib/python3.8/site-packages/gnocchi/common/ceph.py", line 46, in create_rados_connection
    raise ImportError("No module named 'rados'")
ImportError: No module named 'rados'

```
The problem seems to be that the gnocchi upgrade command is run soon as the container is created, so adding the package afterwards doesn't help as the loaded python PID didn't have the library at start time and doesn't reevaluate available libs at runtime. I tried to `kill -HUP` the pid to work around that, but it kills the container outright breaking the deploy

[Buddhika Sanjeewa (bsanjeewa)]

I can confirm that the patch mentioned above by Mark (https://review.opendev.org/c/openstack/kolla/+/791122) resolves the problem.
Patched the kolla repo in the seed, and rebuilt the gnocchi images, and now deployment works.

Deployed with Kayobe/CentOS8-Stream/Wallaby/Source

[Chier Xuefei (ktaog6)]

Hi, Buddhika:

    How to rebuilt the gnocchi images ? Thanks.

[Chier Xuefei (ktaog6)]

Who has this image with bug fixed , can you send me a link ? Thanks .

[Mark Goddard (mgoddard)]

You can rebuild the image with this patch applied to kolla: https://review.opendev.org/c/openstack/kolla/+/791122. It is approved now, and will be backported.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/791122
Committed: https://opendev.org/openstack/kolla/commit/c29f818696c74fd32b929a763695d244e112a33c
Submitter: "Zuul (22348)"
Branch: master

commit c29f818696c74fd32b929a763695d244e112a33c
Author: Mark Goddard <email address hidden>
Date: Thu May 13 09:06:19 2021 +0100

    gnocchi: add python3-rados to gnocchi-base source image

    The binary images include python3-rados as a dependency, but source
    images do not. This change fixes that.

    Change-Id: I2c8cdfcd25856ecdcfd9f302965187b3b62376ad
    Closes-Bug: #1927756

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla/+/801331

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/kolla/+/801332

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/kolla/+/801333

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/801331
Committed: https://opendev.org/openstack/kolla/commit/f0eb095e72b0be9066d584be6968e7ee716e45c4
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f0eb095e72b0be9066d584be6968e7ee716e45c4
Author: Mark Goddard <email address hidden>
Date: Thu May 13 09:06:19 2021 +0100

    gnocchi: add python3-rados to gnocchi-base source image

    The binary images include python3-rados as a dependency, but source
    images do not. This change fixes that.

    Change-Id: I2c8cdfcd25856ecdcfd9f302965187b3b62376ad
    Closes-Bug: #1927756
    (cherry picked from commit c29f818696c74fd32b929a763695d244e112a33c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/801332
Committed: https://opendev.org/openstack/kolla/commit/b9eac37a6fe71307b78727a8d0bf810698457f26
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit b9eac37a6fe71307b78727a8d0bf810698457f26
Author: Mark Goddard <email address hidden>
Date: Thu May 13 09:06:19 2021 +0100

    gnocchi: add python3-rados to gnocchi-base source image

    The binary images include python3-rados as a dependency, but source
    images do not. This change fixes that.

    Change-Id: I2c8cdfcd25856ecdcfd9f302965187b3b62376ad
    Closes-Bug: #1927756
    (cherry picked from commit c29f818696c74fd32b929a763695d244e112a33c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla/+/801333
Committed: https://opendev.org/openstack/kolla/commit/4d122108323482511ffe86ea2a580c26367ebe2c
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 4d122108323482511ffe86ea2a580c26367ebe2c
Author: Mark Goddard <email address hidden>
Date: Thu May 13 09:06:19 2021 +0100

    gnocchi: add python3-rados to gnocchi-base source image

    The binary images include python3-rados as a dependency, but source
    images do not. This change fixes that.

    Change-Id: I2c8cdfcd25856ecdcfd9f302965187b3b62376ad
    Closes-Bug: #1927756
    (cherry picked from commit c29f818696c74fd32b929a763695d244e112a33c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 10.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 12.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 13.0.0.0rc1 release candidate.