Upgrades CI job failing due to service openstack-ceilometer-central not found

[Jose Luis Franco]

Both zuul CI jobs scenario001-multinode-oooq-container-upgrades and scenario002-multinode-oooq-container-upgrades are failing during upgrade due to:

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service openstack-ceilometer-central: host"}

http://logs.openstack.org/99/509199/4/check/legacy-tripleo-ci-centos-7-scenario001-multinode-oooq-container-upgrades/d24ef33/logs/undercloud/home/zuul/failed_upgrade.log.txt.gz

While the service is enabled in the environment file, as well as in the puppet configuration:

https://github.com/openstack/tripleo-heat-templates/blob/master/ci/environments/scenario001-multinode.yaml#L70

"ceilometer_agent_central_enabled": "true",
    "ceilometer_agent_central_short_bootstrap_node_name": "centos-7-rax-ord-0000270320",
    "ceilometer_agent_central_short_node_names": [
        "centos-7-rax-ord-0000270320"

http://logs.openstack.org/99/509199/4/check/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-container-upgrades/2f0128d/logs/subnode-2/etc/puppet/hieradata/all_nodes.json.txt.gz

At the time of upgrading, the openstack-ceilometer-central service is not found in the host and as there are no verifications at the moment in the corresponding docker service template, the upgrade fails:

https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/ceilometer-agent-central.yaml#L131

The job seems to be failing 100% of the times.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/513335

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jose Luis Franco (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/513335

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alex Schultz (<email address hidden>) on branch: master
Review: https://review.openstack.org/507796
Reason: zuul ate it, will restore shortly

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/507796
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=776338d54c030c3a9a02fc0e76ad27f35342fba2
Submitter: Zuul
Branch: master

commit 776338d54c030c3a9a02fc0e76ad27f35342fba2
Author: Jose Luis Franco Arza <email address hidden>
Date: Wed Aug 2 15:28:50 2017 +0200

    Add validation task in docker services [Ceilometer]

    Docker services are missing the pre-upgrade validation task
    in the upgrade_tasks section which verifies if the service
    is running before going on with the upgrade.

    Change-Id: I28488b51e6a2e2d0e0ac41d0d7c35be4edb59cbb
    Partial-Bug: #1704389
    Closes-Bug: #1724478

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/513335
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2100877d64bb55c974a1af374591b10f44b13afd
Submitter: Zuul
Branch: stable/pike

commit 2100877d64bb55c974a1af374591b10f44b13afd
Author: Jose Luis Franco Arza <email address hidden>
Date: Wed Aug 2 15:28:50 2017 +0200

    Add validation task in docker services [Ceilometer]

    Docker services are missing the pre-upgrade validation task
    in the upgrade_tasks section which verifies if the service
    is running before going on with the upgrade.

    (cherry picked from 776338d54c030c3a9a02fc0e76ad27f35342fba2)
    Change-Id: I28488b51e6a2e2d0e0ac41d0d7c35be4edb59cbb
    Partial-Bug: #1704389
    Closes-Bug: #1724478

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.9 release.