octavia is not reporting metrics like lbaasv2

[Wouter van Bommel]

[Environment]

Stein OpenStack
Ubuntu Bionic

[Description]

From my understanding the current Octavia API should be backwards compatible with LBaaS v2, therefore, the current LBaaS v2 pollster [1] should suffice to gather the following meters:

meters:

- network.services.lb.outgoing.bytes
- network.services.lb.incoming.bytes
- network.services.lb.pool
- network.services.lb.listener
- network.services.lb.member
- network.services.lb.health_monitor
- network.services.lb.loadbalancer
- network.services.lb.total.connections
- network.services.lb.active.connections

However, the following warning is noticed when restarted the ceilometer services.

2999:2019-11-07 15:05:41.665 22487 WARNING ceilometer.publisher.gnocchi [-] metric network.services.lb.loadbalancer is not handled by Gnocchi
3467:2019-11-07 15:06:21.830 3916 WARNING ceilometer.publisher.gnocchi [-] metric network.services.lb.loadbalancer is not handled by Gnocchi

In fact, by checking at the gnocchi metric list (https://pastebin.canonical.com/p/zyMTFv8vww/) the network.services.lb
metrics/resources haven't been created.

The reason is that for a gnocchi resource to exist, ceilometer has to create the resource type explicitly.
This action is commanded by the ceilometer gnocchi client [2] and it uses the file /usr/lib/python3/dist-packages/ceilometer/publisher/data/gnocchi_resources.yaml
for the resource definitions, which doesn't includes the lbaasv2 directives.

[Test case]

1) Deploy a gnocchi backed ceilometer unit with octavia deployed.
2) check that lb meters aren't being gathered

$ gnocchi metric list|grep loadbalancer | wc -l
0
3) install the patched version
4) configure all pollsters
$ juju config ceilometer enable-all-pollsters=true
5) sudo systemctl restart ceilometer*
6) sudo ceilometer-upgrade
7) Check that lb metrics are being gathered

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
72

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep network.services
| 3947b249-b1ed-489d-9f51-b7549b5b78ce | ceilometer-low | network.services.lb.loadbalancer | loadbalancer | f97838f1-215d-4fcb-bd78-61e1f9859507 |
| 5527c5b6-d17e-4218-868b-c8d5326575a6 | ceilometer-low | network.services.lb.active.connections | connection | f97838f1-215d-4fcb-bd78-61e1f9859507 |
| 57d63bcb-48e1-47fb-825c-1109ad7d966d | ceilometer-low | network.services.lb.loadbalancer | loadbalancer | 82d129f3-64a5-40d6-a72b-7ae704046176 |
| 5d93c751-0645-4b7b-a315-26b0e96dda3d | ceilometer-low | network.services.lb.outgoing.bytes | B | 82d129f3-64a5-40d6-a72b-7ae704046176 |
| 71ace776-9216-40a4-848c-42e9cc6f93a0 | ceilometer-low | network.services.lb.listener | listener | e7a4dd96-73a1-4aca-946d-8e87ba8f7a37 |
| 738b1dd0-0930-4bfa-b358-1a27fb63121e | ceilometer-low | network.services.lb.total.connections | connection | f97838f1-215d-4fcb-bd78-61e1f9859507 |
| 7b4726f6-9490-4d6c-a586-7e12b738b001 | ceilometer-low | network.services.lb.health_monitor | health_monitor | 3ebc3ac0-8e1c-47d9-8789-bb51e6d29eba |
| 8689e872-c6d0-450f-bb73-d19197c62418 | ceilometer-low | network.services.lb.incoming.bytes | B | 82d129f3-64a5-40d6-a72b-7ae704046176 |
| ae72c5f3-4d8b-400b-ba86-2aaf068c20fc | ceilometer-low | network.services.lb.incoming.bytes | B | f97838f1-215d-4fcb-bd78-61e1f9859507 |
| b59ef5ae-9d71-4884-bb7c-66e0d46f5ecb | ceilometer-low | network.services.lb.outgoing.bytes | B | f97838f1-215d-4fcb-bd78-61e1f9859507 |
| bdf259a1-7471-4c23-ab2b-cac11f522f45 | ceilometer-low | network.services.lb.active.connections | connection | 82d129f3-64a5-40d6-a72b-7ae704046176 |
| e5d2e433-9482-4e2a-9485-e73d7a13271a | ceilometer-low | network.services.lb.listener | listener | c87e0f88-26bf-496f-a882-71b1c67956b4 |
| eb634e4e-8e94-4ee4-8fac-4f4db12d56aa | ceilometer-low | network.services.lb.total.connections | connection | 82d129f3-64a5-40d6-a72b-7ae7040

[Regression Potential]

There is no regression potential identified on this patch as it uses
the new metrics upgrade path and it doesn't
enables any new metrics if the lbaasv2 pollster isn't configured.

[Proposed solution]

Enable the resources on /usr/lib/python3/dist packages/ceilometer/publisher/data/gnocchi_resources.yaml
and modify the gnocchi client provider. I am able to gather the metrics for the octavia load balancers.

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep network.services
| 3947b249-b1ed-489d-9f51-b7549b5b78ce | ceilometer-low | network.services.lb.loadbalancer | loadbalancer | f97838f1-215d-4fcb-bd78-61e1f9859507 |
| 5527c5b6-d17e-4218-868b-c8d5326575a6 | ceilometer-low | network.services.lb.active.connections | connection | f97838f1-215d-4fcb-bd78-61e1f9859507 |
| 57d63bcb-48e1-47fb-825c-1109ad7d966d | ceilometer-low | network.services.lb.loadbalancer | loadbalancer | 82d129f3-64a5-40d6-a72b-7ae704046176 |
| 5d93c751-0645-4b7b-a315-26b0e96dda3d | ceilometer-low | network.services.lb.outgoing.bytes | B | 82d129f3-64a5-40d6-a72b-7ae704046176 |
| 71ace776-9216-40a4-848c-42e9cc6f93a0 | ceilometer-low | network.services.lb.listener | listener | e7a4dd96-73a1-4aca-946d-8e87ba8f7a37 |
| 738b1dd0-0930-4bfa-b358-1a27fb63121e | ceilometer-low | network.services.lb.total.connections | connection | f97838f1-215d-4fcb-bd78-61e1f9859507 |
| 7b4726f6-9490-4d6c-a586-7e12b738b001 | ceilometer-low | network.services.lb.health_monitor | health_monitor | 3ebc3ac0-8e1c-47d9-8789-bb51e6d29eba |
| 8689e872-c6d0-450f-bb73-d19197c62418 | ceilometer-low | network.services.lb.incoming.bytes | B | 82d129f3-64a5-40d6-a72b-7ae704046176 |
| ae72c5f3-4d8b-400b-ba86-2aaf068c20fc | ceilometer-low | network.services.lb.incoming.bytes | B | f97838f1-215d-4fcb-bd78-61e1f9859507 |
| b59ef5ae-9d71-4884-bb7c-66e0d46f5ecb | ceilometer-low | network.services.lb.outgoing.bytes | B | f97838f1-215d-4fcb-bd78-61e1f9859507 |
| bdf259a1-7471-4c23-ab2b-cac11f522f45 | ceilometer-low | network.services.lb.active.connections | connection | 82d129f3-64a5-40d6-a72b-7ae704046176 |
| e5d2e433-9482-4e2a-9485-e73d7a13271a | ceilometer-low | network.services.lb.listener | listener | c87e0f88-26bf-496f-a882-71b1c67956b4 |
| eb634e4e-8e94-4ee4-8fac-4f4db12d56aa | ceilometer-low | network.services.lb.total.connections | connection | 82d129f3-64a5-40d6-a72b-7ae704046176 |
| ef04f494-884a-4097-ba58-74845f5fbf19 | ceilometer-low | network.services.lb.pool | pool | 6c4a5a4f-0455-42e6-8bf4-ed9c24e2c06c |

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi measures show ef04f494-884a-4097-ba58-74845f5fbf19
+---------------------------+-------------+-------+
| timestamp | granularity | value |
+---------------------------+-------------+-------+
| 2019-11-07T18:30:00+00:00 | 300.0 | 1.0 |
| 2019-11-07T18:35:00+00:00 | 300.0 | 1.0 |
| 2019-11-07T18:40:00+00:00 | 300.0 | 1.0 |
+---------------------------+-------------+-------+

[Wouter van Bommel (woutervb)]

Added field medium, as we have a customer waiting for feedback

[Ryan Beisner (1chb1n)]

I'm not certain the upstream projects support this, which is where the feature would need to be added. We will have to research.

[Witek Bedyk (witold-bedyk)]

One way of getting around this is to keep using Ceilometer agent and publishing the measurements to Monasca API [1].

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-system-architecture.html#supported-databases

[Jorge Niedbalski (niedbalski)]

I have modified the case description and i'll provide the changes explained in the proposed solution section.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/693432

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693432
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=cd9e199076b1e3cb8ef6e6bc3e7844b0d23f7e48
Submitter: Zuul
Branch: master

commit cd9e199076b1e3cb8ef6e6bc3e7844b0d23f7e48
Author: Jorge Niedbalski <email address hidden>
Date: Thu Nov 7 18:30:22 2019 -0300

    Add loadbalancer resource type.

    Adds the loadbalancer resource type for usage
    with the network.services.lb.* metrics.

    Change-Id: Ib6e78438c3da0e22d93f720f00cdeadf0ed7a91f
    Closes-Bug: #1848286
    Signed-off-by: Jorge Niedbalski <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/693916

[Jorge Niedbalski (niedbalski)]

Stein debdiff

[Edward Hope-Morley (hopem)]

There are rocky and stein backport submissions @ https://review.opendev.org/#/q/Ib6e78438c3da0e22d93f720f00cdeadf0ed7a91f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/695728
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=18e9bb957b758d4acd82fce1b7eb40af0eaa5b7b
Submitter: Zuul
Branch: stable/stein

commit 18e9bb957b758d4acd82fce1b7eb40af0eaa5b7b
Author: Jorge Niedbalski <email address hidden>
Date: Thu Nov 7 18:30:22 2019 -0300

    Add loadbalancer resource type.

    Adds the loadbalancer resource type for usage
    with the network.services.lb.* metrics.

    Depends-On: https://review.opendev.org/#/c/695855/
    Change-Id: Ib6e78438c3da0e22d93f720f00cdeadf0ed7a91f
    Closes-Bug: #1848286
    Signed-off-by: Jorge Niedbalski <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/695729
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=00ba87d11432ac4ce1b9451a29dcdbeb0a4e973c
Submitter: Zuul
Branch: stable/rocky

commit 00ba87d11432ac4ce1b9451a29dcdbeb0a4e973c
Author: Jorge Niedbalski <email address hidden>
Date: Thu Nov 7 18:30:22 2019 -0300

    Add loadbalancer resource type.

    Adds the loadbalancer resource type for usage
    with the network.services.lb.* metrics.

    Depends-On: https://review.opendev.org/#/c/695855/
    Change-Id: Ib6e78438c3da0e22d93f720f00cdeadf0ed7a91f
    Closes-Bug: #1848286
    Signed-off-by: Jorge Niedbalski <email address hidden>

[Jorge Niedbalski (niedbalski)]

Patches have landed in stable branches. Thanks

[Corey Bryant (corey.bryant)]

Thanks Jorge.

stable/train and stable/queens are still having issues but seeing as this has landed in upstream master (ussuri), stable/stein, and stable/rocky, I think this is ready to move forward with SRUs back to stable/rocky. Upstream stable/queens looks like it may need some work before we can SRU.

Update packages have been uploaded to the following and are awaiting SRU team review:
https://launchpad.net/ubuntu/eoan/+queue?queue_state=1&queue_text=ceilometer
https://launchpad.net/ubuntu/disco/+queue?queue_state=1&queue_text=ceilometer
https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/rocky-staging/+packages?field.name_filter=ceilometer

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceilometer - 1:13.0.0-0ubuntu2

---------------
ceilometer (1:13.0.0-0ubuntu2) focal; urgency=medium

  * d/p/add-loadbalancer-resource-type.patch: Enable load balancer metrics,
    allowing gnocchi to capture measurement data for octavia load balancers
    (LP: #1848286).

 -- Corey Bryant <email address hidden> Wed, 27 Nov 2019 09:14:21 -0500

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693916
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=6f77f2d53e5bcd51da4c46c2d1bec3d708ccaef6
Submitter: Zuul
Branch: stable/train

commit 6f77f2d53e5bcd51da4c46c2d1bec3d708ccaef6
Author: Jorge Niedbalski <email address hidden>
Date: Thu Nov 7 18:30:22 2019 -0300

    Add loadbalancer resource type.

    Adds the loadbalancer resource type for usage
    with the network.services.lb.* metrics.

    Change-Id: Ib6e78438c3da0e22d93f720f00cdeadf0ed7a91f
    Closes-Bug: #1848286
    Signed-off-by: Jorge Niedbalski <email address hidden>
    (cherry picked from commit cd9e199076b1e3cb8ef6e6bc3e7844b0d23f7e48)

[Timo Aaltonen (tjaalton)]

Hello Wouter, or anyone else affected,

Accepted ceilometer into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceilometer/1:13.0.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Wouter, or anyone else affected,

Accepted ceilometer into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ceilometer/1:12.0.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Wouter, or anyone else affected,

Accepted ceilometer into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Wouter, or anyone else affected,

Accepted ceilometer into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Wouter, or anyone else affected,

Accepted ceilometer into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Jorge Niedbalski (niedbalski)]

I've deployed this bundle[0] with proposed and the following configuration:

1) juju config ceilometer enable-all-pollsters=true
2) sudo systemctl restart ceilometer*
3) sudo ceilometer-upgrade

train
-----
ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
0

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
72

stein
-----
ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
0

ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
72

rocky
-----
ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
0
ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
72

eoan
-----
ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
0
ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
72

disco
-----
ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
0
ubuntu@niedbalski-bastion:~/stsstack-bundles/openstack$ gnocchi metric list|grep loadbalancer | wc -l
72

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceilometer - 1:13.0.0-0ubuntu1.1

---------------
ceilometer (1:13.0.0-0ubuntu1.1) eoan; urgency=medium

  * d/gbp.conf: Create stable/train branch.
  * d/p/add-loadbalancer-resource-type.patch: Enable load balancer metrics,
    allowing gnocchi to capture measurement data for octavia load balancers
    (LP: #1848286).

 -- Corey Bryant <email address hidden> Wed, 27 Nov 2019 09:23:05 -0500

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ceilometer has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ceilometer - 1:12.0.0-0ubuntu1.1

---------------
ceilometer (1:12.0.0-0ubuntu1.1) disco; urgency=medium

  * d/gbp.conf: Create stable/stein branch.
  * d/control: Add python-oslo.upgradecheck BD.
  * d/p/add-loadbalancer-resource-type.patch: Enable load balancer metrics,
    allowing gnocchi to capture measurement data for octavia load balancers
    (LP: #1848286).

 -- Corey Bryant <email address hidden> Wed, 27 Nov 2019 09:44:08 -0500

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ceilometer has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ceilometer - 1:13.0.0-0ubuntu1.1~cloud0
---------------

 ceilometer (1:13.0.0-0ubuntu1.1~cloud0) bionic-train; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceilometer (1:13.0.0-0ubuntu1.1) eoan; urgency=medium
 .
   * d/gbp.conf: Create stable/train branch.
   * d/p/add-loadbalancer-resource-type.patch: Enable load balancer metrics,
     allowing gnocchi to capture measurement data for octavia load balancers
     (LP: #1848286).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ceilometer has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ceilometer - 1:12.0.0-0ubuntu1.1~cloud0
---------------

 ceilometer (1:12.0.0-0ubuntu1.1~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ceilometer (1:12.0.0-0ubuntu1.1) disco; urgency=medium
 .
   * d/gbp.conf: Create stable/stein branch.
   * d/control: Add python-oslo.upgradecheck BD.
   * d/p/add-loadbalancer-resource-type.patch: Enable load balancer metrics,
     allowing gnocchi to capture measurement data for octavia load balancers
     (LP: #1848286).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ceilometer has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ceilometer - 1:11.0.1-0ubuntu2~cloud1
---------------

 ceilometer (1:11.0.1-0ubuntu2~cloud1) bionic-rocky; urgency=medium
 .
   * d/p/add-loadbalancer-resource-type.patch: Enable load balancer metrics,
     allowing gnocchi to capture measurement data for octavia load balancers
     (LP: #1848286).

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ceilometer 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ceilometer 12.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ceilometer 13.1.0 release.

[Simon Quigley (tsimonq2)]

Unsubscribing sponsors, as there is nothing left to do. If this was done in error, please resubscribe.