Telemetry containers are unhealthy after fresh deployment

[Emilien Macchi]

Gnocchi, Ceilometer and Sensu containers aren't healthy after a fresh deployment and it's blocking https://review.openstack.org/#/c/569153/ :

http://logs.openstack.org/30/581530/2/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/b5b95b2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-11_01_03_15
http://logs.openstack.org/30/581530/2/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/75d3237/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-11_00_43_50

We need to fix these healthchecks scripts to actually work during a deployment.

[Martin Mágr (mmagr)]

Sensu health check is working as expected actually. The client is not connected to any RabbitMQ instance after CI job deployment ends, so the health check is reporting unhealthy container correctly.

Will fix the rest.

[Alex Schultz (alex-schultz)]

So it sounds like the sensu configuration is not correct for the CI job. We would expect that the container be healthy in CI

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/581759

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/581766

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/581766
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c5b2d08c407cbcb27d4a4037b12195dba5eea9ab
Submitter: Zuul
Branch: master

commit c5b2d08c407cbcb27d4a4037b12195dba5eea9ab
Author: Martin Mágr <email address hidden>
Date: Wed Jul 11 16:33:21 2018 +0200

    Disable sensu-client in CI

    After CI scenario deployment is finished the container ends in unhealthy
    state because sensu-client is not successfully connected to RabbitMQ.
    We will need to implement default connection to overcloud RabbitMQ instance
    for the service to have this service deployed in CI job.

    Change-Id: I1aec0c71a945c06a6d914638b45cae074288a90d
    Closes-Bug: #1781108

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/581759
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=218065b63edd3a173089ea55ab008518b57b4386
Submitter: Zuul
Branch: master

commit 218065b63edd3a173089ea55ab008518b57b4386
Author: Martin Mágr <email address hidden>
Date: Wed Jul 11 13:20:42 2018 +0200

    Fix gnocchi_statsd and ceilometer_agent_ipmi healthcheck

    - statsd is connecting to redis only when it receives a data,
      hence to avoid false negatives we will rather test if the service
      is listening on proper port

    - ceilometer_agent_ipmi connects to Redis instead of Rabbit for coordination
      now.

    Closes-Bug: #1781108
    Change-Id: I4414bac82981c30ba6b7c1c8b406baa7a0900e5a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/583714

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/583714
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=a70d31c236ab460b4bc1a561c96cfe9e262dd020
Submitter: Zuul
Branch: stable/queens

commit a70d31c236ab460b4bc1a561c96cfe9e262dd020
Author: Martin Mágr <email address hidden>
Date: Wed Jul 11 13:20:42 2018 +0200

    Fix gnocchi_statsd and ceilometer_agent_ipmi healthcheck

    - statsd is connecting to redis only when it receives a data,
      hence to avoid false negatives we will rather test if the service
      is listening on proper port

    - ceilometer_agent_ipmi connects to Redis instead of Rabbit for coordination
      now.

    Closes-Bug: #1781108
    Change-Id: I4414bac82981c30ba6b7c1c8b406baa7a0900e5a
    (cherry picked from commit 218065b63edd3a173089ea55ab008518b57b4386)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 9.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b4 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 8.6.4 release.