charm should make it more obvious that gnocchi is not yet initialised

[Edward Hope-Morley]

It is currently mandatory to manually run the ceilometer-upgrade action post deployment in order for gnocchi to be populated with ceilometer resources (and be able to do anything). The actions.yaml does say "This action must be run post deployment when ceilometer is related to gnocchi" but that's only useful if you think to look there. Perhaps it would be better to do one or more of the following:
  * set a charm status saying that gnocchi is not ready yet until this action has been run
  * have the charm automatically run the upgrade (using goal state info to know when gnocchi is ready)
  * rename the action to something like "init-gnocchi" so that it sound less like something that needs to be run only when upgrading to gnocchi from mongo.

[David Ames (thedac)]

Triage:

First set ceilometer workload status blocked if the ceilometer-upgrade action has not run.

Secondly, explore goal state on both gnocchi and ceilometer to revisit automatically running the upgrade code path.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/633848

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633848
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=e2ca8f06de0d727cb9b196ad6b10d021071c9948
Submitter: Zuul
Branch: master

commit e2ca8f06de0d727cb9b196ad6b10d021071c9948
Author: David Ames <email address hidden>
Date: Tue Jan 29 23:16:23 2019 +0000

    Status blocked until ceilometer-upgrade action

    Currently it is unclear that ceilometer is not actually ready when it
    is related to gnocchi until the ceilometer-upgrade action has run. The
    status will show active Unit is ready when it is not.

    When gnocchi is related, check that the ceilometer-upgrade action has
    been run. If not set workload status to prompt the administator to run
    the action.

    Please merge the charm-helpers change first:
    https://github.com/juju/charm-helpers/pull/270

    Partial-Bug: #1811108
    Change-Id: Id778ed3f59e4bcde9c6586aad46be2b5b9ccb531

[David Ames (thedac)]

The workload status will now accurately inform the end user for the need to run the ceilometer-upgrade action.

I am going to lower the priority of this bug but keep it open for the potential to utilize goal state to improve on the solution.

[Edward Hope-Morley (hopem)]

@thedac i think this patch might needs fixing because I just upgraded my ceilometer after having run the action and it displays the blocked status:

ubuntu@hopem-bastion:~/stsstack-bundles/openstack$ juju show-action-status
actions:
- action: ceilometer-upgrade
  completed at: "2019-02-01 12:06:24"
  id: 5c16cdee-4215-490c-81a3-5c84b2a90b9d
  status: completed
  unit: ceilometer/0
ubuntu@hopem-bastion:~/stsstack-bundles/openstack$ juju status ceilometer
Model Controller Cloud/Region Version SLA Timestamp
sf00213107-dxcit hopem stsstack/stsstack 2.4.7 unsupported 13:20:59Z

App Version Status Scale Charm Store Rev OS Notes
ceilometer 10.0.1 blocked 1 ceilometer jujucharms 325 ubuntu

Unit Workload Agent Machine Public address Ports Message
ceilometer/0* blocked idle 1 10.5.0.7 Run the ceilometer-upgrade action to initialize ceilometer and gnocchi

Machine State DNS Inst id Series AZ Message
1 started 10.5.0.7 e5f43020-22c8-48ad-b8d0-7ae131869347 bionic nova ACTIVE

[Edward Hope-Morley (hopem)]

Note that running the action again does clear the blocked message.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/635639

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/635639
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=0a59a19ad20221b088843458e3b1cc8c3cdca6c3
Submitter: Zuul
Branch: master

commit 0a59a19ad20221b088843458e3b1cc8c3cdca6c3
Author: David Ames <email address hidden>
Date: Thu Feb 7 19:48:10 2019 +0000

    Use leader set for ceilometer-upgrade checks

    The previous attempt at addressing Bug#1811108 was a bit naive.
    It was not HA aware and it failed to handle upgrade-charm.

    This change moves from kv to leader-set as only the leader needs to run
    the action and then inform the other nodes.

    Note: on upgrade-charm we must assume the deployment has run
    ceilometer-upgrade as there is no mechanism to determine this
    independently. If we do not do this, the charm is in blocked state after
    charm-upgrade. A warning is logged that this assumption has occurred.

    Partial-Bug: #1811108
    Change-Id: Idcc26df53542e78f0671942c99edfcbf61eccf6c

[David Ames (thedac)]

Both https://review.openstack.org/633848 and https://review.openstack.org/635639 landed.