Ceilometer

Bug #1777607
Comment #3

Comment 3 for bug 1777607

Revision history for this message

Eric Miller (erickmiller) wrote on 2019-08-30:

As liyi pointed out, the problem will also occur when the flavor is private.

We are using Kolla Ansible for deployments. This deployment uses Kolla Ansible 7.0.2 (Rocky).

We also use a custom Gnocchi archive policy for all metrics, so the default policy is not used.

We ran into a similar issue, but this is the process we went through:
- Start with existing cluster with public and private flavors defined
- Existing instances exist with both public and private flavors
- Have a need to change the Gnocchi archive-policy drastically, so re-install Gnocchi using:
    - Stop all Ceilometer and Gnocchi containers
    - Drop gnocchi MySQL table
    - Purge all objects from gnocchi pool in Ceph
    - "Deploy" Ceilometer and Gnocchi with Kolla Ansible so MySQL table is re-created and Ceilometer bootstrap is run to create Gnocchi resource types

At this point, numerous messages are queued in RabbitMQ and Gnocchi isn't responding to these since the archive policy for our metrics does not exist yet.

We then create our custom archive policy and RabbitMQ messages start getting processed.

Cross checking the instances with the Gnocchi data shows that instances with private flavors do not have metrics. Plus, we get the error listed in this bug report (required key not provided @ data[u'flavor_id']) logged in the following log file on all controllers:
/var/lib/docker/volumes/kolla_logs/_data/ceilometer/ceilometer-agent-notification.log

What is odd, though, is that after some time has passed (15 minutes?), SOME of the instances with private flavors now have metrics, and the error above is no longer being logged for these instances.

However, other instances never receive metrics and the above error persists. We have found no differences in configurations of VMs, hosts, etc. for those that have the error and those that don't. We have made sure that all instances that have the error have a valid flavor that has NOT been deleted.

So, while this is similar to this bug report, it seems something else is happening here where flavor IDs are not being found during the discovery process. Maybe a random ordering issue and truncation of a list?

Eric

As liyi pointed out, the problem will also occur when the flavor is private.

We are using Kolla Ansible for deployments.  This deployment uses Kolla Ansible 7.0.2 (Rocky).

We also use a custom Gnocchi archive policy for all metrics, so the default policy is not used.

We ran into a similar issue, but this is the process we went through:
 - Start with existing cluster with public and private flavors defined
 - Existing instances exist with both public and private flavors
 - Have a need to change the Gnocchi archive-policy drastically, so re-install Gnocchi using:
    - Stop all Ceilometer and Gnocchi containers
    - Drop gnocchi MySQL table
    - Purge all objects from gnocchi pool in Ceph
    - "Deploy" Ceilometer and Gnocchi with Kolla Ansible so MySQL table is re-created and Ceilometer bootstrap is run to create Gnocchi resource types

At this point, numerous messages are queued in RabbitMQ and Gnocchi isn't responding to these since the archive policy for our metrics does not exist yet.

We then create our custom archive policy and RabbitMQ messages start getting processed.

Cross checking the instances with the Gnocchi data shows that instances with private flavors do not have metrics.  Plus, we get the error listed in this bug report (required key not provided @ data[u'flavor_id']) logged in the following log file on all controllers:
/var/lib/docker/volumes/kolla_logs/_data/ceilometer/ceilometer-agent-notification.log

What is odd, though, is that after some time has passed (15 minutes?), SOME of the instances with private flavors now have metrics, and the error above is no longer being logged for these instances.

However, other instances never receive metrics and the above error persists.  We have found no differences in configurations of VMs, hosts, etc. for those that have the error and those that don't.  We have made sure that all instances that have the error have a valid flavor that has NOT been deleted.

So, while this is similar to this bug report, it seems something else is happening here where flavor IDs are not being found during the discovery process.  Maybe a random ordering issue and truncation of a list?

Eric