Comment 21 for bug 1749280

Revision history for this message
David Ames (thedac) wrote :

Update: This bug has really been multiple race conditions.
There was a race with the Database being synced in gnocchi.
There was a race with hacluster being formed with gnocchi.
All of these present when ceilometer runs ceilometer-upgrade.

Now in what I hope is the final race condition, there is a race with keystone catalog and the availability of the metric catalog entry.

Now gnocchi has synced its db, has a CRM cluster and receives information from keystone. Gnocchi informs ceilometer. Ceilometer attempts to run ceilometer-upgrade but keystone has not yet put the metric service in the catalog.

CRITICAL ceilometer [-] EndpointNotFound: internalURL endpoint for metric service not found

Next areas to investigate:

The interface-keystone has multiple data_complete options. Is it possible we have some but not all of the data from kesytone. i.e. identity-service.available vs identity-service.available.auth

A cursory look at keystone seems that it registers the catalog entry contemporaneously with setting relation data. Need to confirm this is the case.

Note: Keystone is very busy executing hooks.
Could there be a delay between running the commands to publish the catalog entry and when it is actually available?