gnocchi create resource error when flavor deleted

Bug #1777607 reported by liyi
34
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Ceilometer
In Progress
Undecided
Unassigned

Bug Description

When a server created, I delete the related flavor, and the gnocchi can not discover the server.

Re-produce steps:
1. create a server named server_A with flavor named flavor_A;
2. waiting for server_A creation success;
3. delete flavor_A after server_A success;
4. wait for ceilometer to discover server_A, and check the logs, we will find follow error:

2018-06-19 15:18:04.540 59213 ERROR ceilometer.dispatcher.gnocchi [-] Error creating resource ee9d684b-003f-4874-87ec-f9db5908cee9: Invalid input: required key not provided @ data[u'flavor_id'] (HTTP 400): BadRequest: Invalid input: required key not provided @ data[u'flavor_id'] (HTTP 400)

liyi (whyliyi)
Changed in ceilometer:
assignee: nobody → liyi (whyliyi)
Revision history for this message
liyi (whyliyi) wrote :

The same problem will occur when flavor_A is private. Because we must use is_public param. refer to: https://developer.openstack.org/api-ref/compute/#list-flavors

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ceilometer (master)

Fix proposed to branch: master
Review: https://review.openstack.org/576460

Changed in ceilometer:
status: New → In Progress
Revision history for this message
Eric Miller (erickmiller) wrote :

As liyi pointed out, the problem will also occur when the flavor is private.

We are using Kolla Ansible for deployments. This deployment uses Kolla Ansible 7.0.2 (Rocky).

We also use a custom Gnocchi archive policy for all metrics, so the default policy is not used.

We ran into a similar issue, but this is the process we went through:
 - Start with existing cluster with public and private flavors defined
 - Existing instances exist with both public and private flavors
 - Have a need to change the Gnocchi archive-policy drastically, so re-install Gnocchi using:
    - Stop all Ceilometer and Gnocchi containers
    - Drop gnocchi MySQL table
    - Purge all objects from gnocchi pool in Ceph
    - "Deploy" Ceilometer and Gnocchi with Kolla Ansible so MySQL table is re-created and Ceilometer bootstrap is run to create Gnocchi resource types

At this point, numerous messages are queued in RabbitMQ and Gnocchi isn't responding to these since the archive policy for our metrics does not exist yet.

We then create our custom archive policy and RabbitMQ messages start getting processed.

Cross checking the instances with the Gnocchi data shows that instances with private flavors do not have metrics. Plus, we get the error listed in this bug report (required key not provided @ data[u'flavor_id']) logged in the following log file on all controllers:
/var/lib/docker/volumes/kolla_logs/_data/ceilometer/ceilometer-agent-notification.log

What is odd, though, is that after some time has passed (15 minutes?), SOME of the instances with private flavors now have metrics, and the error above is no longer being logged for these instances.

However, other instances never receive metrics and the above error persists. We have found no differences in configurations of VMs, hosts, etc. for those that have the error and those that don't. We have made sure that all instances that have the error have a valid flavor that has NOT been deleted.

So, while this is similar to this bug report, it seems something else is happening here where flavor IDs are not being found during the discovery process. Maybe a random ordering issue and truncation of a list?

Eric

liyi (whyliyi)
Changed in ceilometer:
assignee: liyi (whyliyi) → nobody
Revision history for this message
Laszlo Budai (laszlo-budai) wrote :

Hi,

we have this issue with private flavours in openstack queens. Is there any solution/workaround for it?

Thank you,
Laszlo

Revision history for this message
Laszlo Budai (laszlo-budai) wrote :

Hi,

we have this patch applied https://review.opendev.org/#/c/576460/3/ceilometer/compute/discovery.py, but still got the same errors for private flavors. Do you have any idea how can we overcome this issue?

Thank you,
Laszlo

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ceilometer 16.0.0.0rc1

This issue was fixed in the openstack/ceilometer 16.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.