metering_secret inconsistent

[Billy Olsen]

The metering_secret is inconsistent across units of the ceilometer service, which causes the ceilometer colelctor to fail with errors such as:

016-02-01 06:57:01.166 28406 WARNING ceilometer.dispatcher.database [-] message signature invalid, discarding message:

Checking the metering_secret stored on the units, I see:

ubuntu@billy-olsen-bastion:~/openstack-charm-testing> juju run --service ceilometer 'sudo grep metering_secret /etc/ceilometer/ceilometer.conf'
- MachineId: "1"
  Stdout: |
    metering_secret = dc991082-4e6d-4e05-8219-e390565ae0ca
  UnitId: ceilometer/0
- MachineId: "23"
  Stdout: |
    metering_secret = 899e31f2-1dea-44fb-a981-a3369810e12a
  UnitId: ceilometer/1
- MachineId: "24"
  Stdout: |
    metering_secret = e551c3a6-7c49-4519-af75-2e045c8d7d23
  UnitId: ceilometer/2

This appears to be fallout from the leadership election changes. The value is stored in the leadership settings, but its not honored in the peer units of the leader:

ubuntu@billy-olsen-bastion:~/openstack-charm-testing> juju run --service ceilometer 'is-leader'
- MachineId: "1"
  Stdout: |
    True
  UnitId: ceilometer/0
- MachineId: "23"
  Stdout: |
    False
  UnitId: ceilometer/1
- MachineId: "24"
  Stdout: |
    False
  UnitId: ceilometer/2
ubuntu@billy-olsen-bastion:~/openstack-charm-testing> juju run --unit ceilometer/0 'leader-get'
shared_secret: dc991082-4e6d-4e05-8219-e390565ae0ca

[Billy Olsen (billy-olsen)]

Note, a work around is the following:

juju run --service ceilometer 'leader-get shared_secret | sudo tee /etc/ceilometer/secret.txt'

and then run a config changed hook to update the ceilometer.conf settings, (e.g. juju set ceilometer debug=True, then juju set ceilometer debug=False).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable
Review: https://review.openstack.org/292641

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/292641
Committed: https://git.openstack.org/cgit/openstack/charm-ceilometer/commit/?id=09100f95b6d9ad31818ca5bd536109b40bf966e0
Submitter: Jenkins
Branch: stable

commit 09100f95b6d9ad31818ca5bd536109b40bf966e0
Author: Billy Olsen <email address hidden>
Date: Wed Feb 3 17:09:08 2016 -0700

    Use peer_retrieve to get secrets from other nodes.

    The secret sharing previously worked because it used the peer storage
    prior to leadership election due to the direct usage of relation_get
    rather than using the retrieval methods provided by the peer storage
    framework.

    Closes Bug: #1541618

    (cherry picked from commit fafc1d4ff52eb04779f0109b8b096e8e84af7f23)

    Change-Id: I9ed88f17bf8659eeaeab4b86fea5c7710611a779