Update: This bug has really been multiple race conditions.
There was a race with the Database being synced in gnocchi.
There was a race with hacluster being formed with gnocchi.
All of these present when ceilometer runs ceilometer-upgrade.
Now in what I hope is the final race condition, there is a race with keystone catalog and the availability of the metric catalog entry.
Now gnocchi has synced its db, has a CRM cluster and receives information from keystone. Gnocchi informs ceilometer. Ceilometer attempts to run ceilometer-upgrade but keystone has not yet put the metric service in the catalog.
CRITICAL ceilometer [-] EndpointNotFound: internalURL endpoint for metric service not found
Next areas to investigate:
The interface-keystone has multiple data_complete options. Is it possible we have some but not all of the data from kesytone. i.e. identity-service.available vs identity-service.available.auth
A cursory look at keystone seems that it registers the catalog entry contemporaneously with setting relation data. Need to confirm this is the case.
Note: Keystone is very busy executing hooks.
Could there be a delay between running the commands to publish the catalog entry and when it is actually available?
Update: This bug has really been multiple race conditions.
There was a race with the Database being synced in gnocchi.
There was a race with hacluster being formed with gnocchi.
All of these present when ceilometer runs ceilometer-upgrade.
Now in what I hope is the final race condition, there is a race with keystone catalog and the availability of the metric catalog entry.
Now gnocchi has synced its db, has a CRM cluster and receives information from keystone. Gnocchi informs ceilometer. Ceilometer attempts to run ceilometer-upgrade but keystone has not yet put the metric service in the catalog.
CRITICAL ceilometer [-] EndpointNotFound: internalURL endpoint for metric service not found
Next areas to investigate:
The interface-keystone has multiple data_complete options. Is it possible we have some but not all of the data from kesytone. i.e. identity- service. available vs identity- service. available. auth
A cursory look at keystone seems that it registers the catalog entry contemporaneously with setting relation data. Need to confirm this is the case.
Note: Keystone is very busy executing hooks.
Could there be a delay between running the commands to publish the catalog entry and when it is actually available?