master: gnocchiclient.exceptions.ResourceTypeNotFound: Resource type image does not exist (HTTP 404)

Bug #1922282 reported by wes hayutin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Unassigned

Bug Description

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/9815483/logs/undercloud/var/log/containers/ceilometer/ceilometer-upgrade.log.txt.gz

2021-04-01 10:28:03.947 13 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python3.6/site-packages/ceilometer/cmd/storage.py:45
2021-04-01 10:28:05.492 13 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.Forbidden: Forbidden (HTTP 403)
2021-04-01 10:28:05.492 13 ERROR ceilometer Traceback (most recent call last):
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/ceilometer/gnocchi_client.py", line 266, in upgrade_resource_types
2021-04-01 10:28:05.492 13 ERROR ceilometer gnocchi.resource_type.get(name=name)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/resource_type.py", line 44, in get
2021-04-01 10:28:05.492 13 ERROR ceilometer headers={'Content-Type': "application/json"}).json()
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/base.py", line 37, in _get
2021-04-01 10:28:05.492 13 ERROR ceilometer return self.client.api.get(*args, **kwargs)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 395, in get
2021-04-01 10:28:05.492 13 ERROR ceilometer return self.request(url, 'GET', **kwargs)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/client.py", line 52, in request
2021-04-01 10:28:05.492 13 ERROR ceilometer raise exceptions.from_response(resp, method)
2021-04-01 10:28:05.492 13 ERROR ceilometer gnocchiclient.exceptions.ResourceTypeNotFound: Resource type image does not exist (HTTP 404)
2021-04-01 10:28:05.492 13 ERROR ceilometer
2021-04-01 10:28:05.492 13 ERROR ceilometer During handling of the above exception, another exception occurred:
2021-04-01 10:28:05.492 13 ERROR ceilometer
2021-04-01 10:28:05.492 13 ERROR ceilometer Traceback (most recent call last):
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2021-04-01 10:28:05.492 13 ERROR ceilometer sys.exit(upgrade())
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/ceilometer/cmd/storage.py", line 60, in upgrade
2021-04-01 10:28:05.492 13 ERROR ceilometer )(gnocchi_client.upgrade_resource_types, conf)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 409, in call
2021-04-01 10:28:05.492 13 ERROR ceilometer do = self.iter(retry_state=retry_state)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 356, in iter
2021-04-01 10:28:05.492 13 ERROR ceilometer return fut.result()
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
2021-04-01 10:28:05.492 13 ERROR ceilometer return self.__get_result()
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-04-01 10:28:05.492 13 ERROR ceilometer raise self._exception
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 412, in call
2021-04-01 10:28:05.492 13 ERROR ceilometer result = fn(*args, **kwargs)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/ceilometer/gnocchi_client.py", line 269, in upgrade_resource_types
2021-04-01 10:28:05.492 13 ERROR ceilometer gnocchi.resource_type.create(resource_type=rt)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/resource_type.py", line 35, in create
2021-04-01 10:28:05.492 13 ERROR ceilometer data=ujson.dumps(resource_type)).json()
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/base.py", line 41, in _post
2021-04-01 10:28:05.492 13 ERROR ceilometer return self.client.api.post(*args, **kwargs)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post
2021-04-01 10:28:05.492 13 ERROR ceilometer return self.request(url, 'POST', **kwargs)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/client.py", line 52, in request
2021-04-01 10:28:05.492 13 ERROR ceilometer raise exceptions.from_response(resp, method)
2021-04-01 10:28:05.492 13 ERROR ceilometer gnocchiclient.exceptions.Forbidden: Forbidden (HTTP 403)

Revision history for this message
wes hayutin (weshayutin) wrote :
Revision history for this message
Takashi Kajinami (kajinamit) wrote :
Revision history for this message
Takashi Kajinami (kajinamit) wrote :

It seems that /etc/gnocchi/policy.yaml is not generated properly and that seems to be the cause why Gnocchi returnes 403.
We need to check why the logic to convert policy.json to policy.yaml is not working as expected.

Revision history for this message
Paul Leimer (pleimer) wrote :

It looks like your patch adding the 'exec' tag for puppet tripleo fixed the issue, now scenario002 passes. Scenario001 is still failing for a different reason. Puppet fails because a ceph directory doesn't exit.

```
2021-04-06T16:10:33.498854609+00:00 stderr F <13>Apr 6 16:10:33 puppet-user: Notice: /Stage[main]/Apache::Mod::Status/File[status.conf]/ensure: defined content as '{md5}fa95c477a2085c1f7f17ee5f8eccfb90'

2021-04-06T16:10:33.544295196+00:00 stderr F <13>Apr 6 16:10:33 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[exec-setfacl-openstack-gnocchi]/returns: setfacl: /etc/ceph/ceph.client.openstack.keyring: No such file or directory

2021-04-06T16:10:33.547817997+00:00 stderr F <13>Apr 6 16:10:33 puppet-user: Error: 'setfacl -m u:gnocchi:r-- /etc/ceph/ceph.client.openstack.keyring' returned 1 instead of one of [0]

2021-04-06T16:10:33.548188877+00:00 stderr F <13>Apr 6 16:10:33 puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[exec-setfacl-openstack-gnocchi]/returns: change from 'notrun' to ['0'] failed: 'setfacl -m u:gnocchi:r--
```

Is this even necessary for these jobs?

Revision history for this message
Takashi Kajinami (kajinamit) wrote :

So the problem was that current puppet-gnocchi requires exec resource so that policy.yaml is properly generated.

However we didn't enable exec resource during gnocchi puppet thus the file was not generated properly.

The failure mentioned in #4 was caused by the old logic to set facl on ceph keyrings, but since that logic have been disabled since we switch to containerized services, then we can just remove these useless tasks so that we can use the exec resource only for policy file generation.

Revision history for this message
Paul Leimer (pleimer) wrote :

thanks for addressing this, Takashi

Changed in tripleo:
status: Triaged → Fix Committed
Revision history for this message
Matthias Runge (mrunge) wrote :

Can we get some clarification here what was changed/fixed or what to pull in?

Revision history for this message
Matthias Runge (mrunge) wrote :
Revision history for this message
Takashi Kajinami (kajinamit) wrote :

This issue was fixed by the two patches in the following link.
 https://review.opendev.org/q/topic:bug/1922282

Revision history for this message
Matthias Runge (mrunge) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 14.1.0

This issue was fixed in the openstack/tripleo-heat-templates 14.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-quickstart (master)

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/785089

wes hayutin (weshayutin)
Changed in tripleo:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers