master: gnocchiclient.exceptions.ResourceTypeNotFound: Resource type image does not exist (HTTP 404)

[wes hayutin]

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/9815483/logs/undercloud/var/log/containers/ceilometer/ceilometer-upgrade.log.txt.gz

2021-04-01 10:28:03.947 13 DEBUG ceilometer.cmd.storage [-] Upgrading Gnocchi resource types upgrade /usr/lib/python3.6/site-packages/ceilometer/cmd/storage.py:45
2021-04-01 10:28:05.492 13 CRITICAL ceilometer [-] Unhandled error: gnocchiclient.exceptions.Forbidden: Forbidden (HTTP 403)
2021-04-01 10:28:05.492 13 ERROR ceilometer Traceback (most recent call last):
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/ceilometer/gnocchi_client.py", line 266, in upgrade_resource_types
2021-04-01 10:28:05.492 13 ERROR ceilometer gnocchi.resource_type.get(name=name)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/resource_type.py", line 44, in get
2021-04-01 10:28:05.492 13 ERROR ceilometer headers={'Content-Type': "application/json"}).json()
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/base.py", line 37, in _get
2021-04-01 10:28:05.492 13 ERROR ceilometer return self.client.api.get(*args, **kwargs)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 395, in get
2021-04-01 10:28:05.492 13 ERROR ceilometer return self.request(url, 'GET', **kwargs)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/client.py", line 52, in request
2021-04-01 10:28:05.492 13 ERROR ceilometer raise exceptions.from_response(resp, method)
2021-04-01 10:28:05.492 13 ERROR ceilometer gnocchiclient.exceptions.ResourceTypeNotFound: Resource type image does not exist (HTTP 404)
2021-04-01 10:28:05.492 13 ERROR ceilometer
2021-04-01 10:28:05.492 13 ERROR ceilometer During handling of the above exception, another exception occurred:
2021-04-01 10:28:05.492 13 ERROR ceilometer
2021-04-01 10:28:05.492 13 ERROR ceilometer Traceback (most recent call last):
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/bin/ceilometer-upgrade", line 10, in <module>
2021-04-01 10:28:05.492 13 ERROR ceilometer sys.exit(upgrade())
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/ceilometer/cmd/storage.py", line 60, in upgrade
2021-04-01 10:28:05.492 13 ERROR ceilometer )(gnocchi_client.upgrade_resource_types, conf)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 409, in call
2021-04-01 10:28:05.492 13 ERROR ceilometer do = self.iter(retry_state=retry_state)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 356, in iter
2021-04-01 10:28:05.492 13 ERROR ceilometer return fut.result()
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 425, in result
2021-04-01 10:28:05.492 13 ERROR ceilometer return self.__get_result()
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
2021-04-01 10:28:05.492 13 ERROR ceilometer raise self._exception
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/tenacity/__init__.py", line 412, in call
2021-04-01 10:28:05.492 13 ERROR ceilometer result = fn(*args, **kwargs)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/ceilometer/gnocchi_client.py", line 269, in upgrade_resource_types
2021-04-01 10:28:05.492 13 ERROR ceilometer gnocchi.resource_type.create(resource_type=rt)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/resource_type.py", line 35, in create
2021-04-01 10:28:05.492 13 ERROR ceilometer data=ujson.dumps(resource_type)).json()
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/v1/base.py", line 41, in _post
2021-04-01 10:28:05.492 13 ERROR ceilometer return self.client.api.post(*args, **kwargs)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 401, in post
2021-04-01 10:28:05.492 13 ERROR ceilometer return self.request(url, 'POST', **kwargs)
2021-04-01 10:28:05.492 13 ERROR ceilometer File "/usr/lib/python3.6/site-packages/gnocchiclient/client.py", line 52, in request
2021-04-01 10:28:05.492 13 ERROR ceilometer raise exceptions.from_response(resp, method)
2021-04-01 10:28:05.492 13 ERROR ceilometer gnocchiclient.exceptions.Forbidden: Forbidden (HTTP 403)

[wes hayutin (weshayutin)]

fails both scenario001 and scenario002

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/9815483/logs/undercloud/var/log/extra/errors.txt.txt.gz

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario002-standalone-master/54ea688/logs/undercloud/var/log/containers/ceilometer/

[Takashi Kajinami (kajinamit)]

Looking at the ceilometer.conf in that job parameters under service_credentials section are set as expected.
 https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/9815483/logs/undercloud/var/lib/config-data/puppet-generated/ceilometer/etc/ceilometer/ceilometer.conf.txt.gz

So I don't think that the change in puppet-ceilometer[1] is the cause.

 [1] https://review.opendev.org/c/openstack/puppet-ceilometer/+/775730

[Takashi Kajinami (kajinamit)]

It seems that /etc/gnocchi/policy.yaml is not generated properly and that seems to be the cause why Gnocchi returnes 403.
We need to check why the logic to convert policy.json to policy.yaml is not working as expected.

[Paul Leimer (pleimer)]

It looks like your patch adding the 'exec' tag for puppet tripleo fixed the issue, now scenario002 passes. Scenario001 is still failing for a different reason. Puppet fails because a ceph directory doesn't exit.

```
2021-04-06T16:10:33.498854609+00:00 stderr F <13>Apr 6 16:10:33 puppet-user: Notice: /Stage[main]/Apache::Mod::Status/File[status.conf]/ensure: defined content as '{md5}fa95c477a2085c1f7f17ee5f8eccfb90'

2021-04-06T16:10:33.544295196+00:00 stderr F <13>Apr 6 16:10:33 puppet-user: Notice: /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[exec-setfacl-openstack-gnocchi]/returns: setfacl: /etc/ceph/ceph.client.openstack.keyring: No such file or directory

2021-04-06T16:10:33.547817997+00:00 stderr F <13>Apr 6 16:10:33 puppet-user: Error: 'setfacl -m u:gnocchi:r-- /etc/ceph/ceph.client.openstack.keyring' returned 1 instead of one of [0]

2021-04-06T16:10:33.548188877+00:00 stderr F <13>Apr 6 16:10:33 puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Gnocchi::Api/Exec[exec-setfacl-openstack-gnocchi]/returns: change from 'notrun' to ['0'] failed: 'setfacl -m u:gnocchi:r--
```

Is this even necessary for these jobs?

[Takashi Kajinami (kajinamit)]

So the problem was that current puppet-gnocchi requires exec resource so that policy.yaml is properly generated.

However we didn't enable exec resource during gnocchi puppet thus the file was not generated properly.

The failure mentioned in #4 was caused by the old logic to set facl on ceph keyrings, but since that logic have been disabled since we switch to containerized services, then we can just remove these useless tasks so that we can use the exec resource only for policy file generation.

[Paul Leimer (pleimer)]

thanks for addressing this, Takashi

[Matthias Runge (mrunge)]

Can we get some clarification here what was changed/fixed or what to pull in?

[Matthias Runge (mrunge)]

Is are these all required patches? https://review.opendev.org/q/topic:%22service_credentials%22+(status:open%20OR%20status:merged)

[Takashi Kajinami (kajinamit)]

This issue was fixed by the two patches in the following link.
 https://review.opendev.org/q/topic:bug/1922282

[Matthias Runge (mrunge)]

My comment #8 was wrong, these patches are required: https://review.opendev.org/q/topic:%22bug%252F1922282%22+(status:open%20OR%20status:merged)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 14.1.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "chandan kumar <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/785089