overcloud upgrade from Ocata to Pike with containerized services failed due to "Could not find the requested service Could not find the requested service openstack-ceilometer-agent-central: cannot disable: cannot disable"

[Artem Hrechanychenko]

Description
===========
I tried to upgrade Ocata cluster to Pike with containerized services.
overcloud upgrade failed:

2017-06-06 09:04:32Z [overcloud-AllNodesDeploySteps-44wkajn6npfp.ControllerUpgrade_Step2.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2017-06-06 09:04:32Z [overcloud-AllNodesDeploySteps-44wkajn6npfp.ControllerUpgrade_Step2]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2017-06-06 09:04:32Z [overcloud-AllNodesDeploySteps-44wkajn6npfp.ControllerUpgrade_Step2]: CREATE_FAILED Error: resources.ControllerUpgrade_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2017-06-06 09:04:32Z [overcloud-AllNodesDeploySteps-44wkajn6npfp]: UPDATE_FAILED Error: resources.ControllerUpgrade_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2017-06-06 09:04:32Z [AllNodesDeploySteps]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerUpgrade_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2017-06-06 09:04:33Z [overcloud]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerUpgrade_Step2.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2

 Stack overcloud UPDATE_FAILED

overcloud.AllNodesDeploySteps.ControllerUpgrade_Step2.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 71d0e159-4d88-41c1-8056-31086382c53f
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
    changed: [localhost] => (item=openstack-swift-object-updater)
    changed: [localhost] => (item=openstack-swift-object)

    TASK [Stop and disable ceilometer agent central service] ***********************
    fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service openstack-ceilometer-agent-central: cannot disable"}
     to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/027e6b33-9fc9-43e7-98ab-58ace76a0f42_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=28 changed=23 unreachable=0 failed=1

    (truncated, view all with --long)
  deploy_stderr: |

Heat Stack update failed.
Heat Stack update failed.

Steps to reproduce
==================
1)Install undercloud v.Ocata using tripleo-quickstart:
bash quickstart.sh --release ocata $HOST

2)deploy overcloud with workaround for https://bugs.launchpad.net/tripleo/+bug/1696052:
 openstack overcloud deploy \
    --libvirt-type qemu \
    --ntp-server clock.redhat.com \
    --control-scale 1 \
    --control-flavor oooq_control \
    --compute-flavor oooq_compute \
    --templates $THT \
    -e $THT/environments/low-memory-usage.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
-e /home/stack/network-environment.yaml \

3) perform upgrade procedure for undercloud
### UPGRADE UNDERCLOUD ###

# master repos
sudo curl -L -o /etc/yum.repos.d/delorean.repo https://trunk.rdoproject.org/centos7-master/current-passed-ci/delorean.repo

sudo curl -L -o /etc/yum.repos.d/delorean-current.repo https://trunk.rdoproject.org/centos7/current/delorean.repo
sudo sed -i 's/\[delorean\]/\[delorean-current\]/' /etc/yum.repos.d/delorean-current.repo
sudo /bin/bash -c "cat <<EOF>>/etc/yum.repos.d/delorean-current.repo

includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common*,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,puppet-*
EOF"
sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo https://trunk.rdoproject.org/centos7/delorean-deps.repo

sudo systemctl stop openstack-*
sudo systemctl stop neutron-*
sudo systemctl stop httpd
sudo yum -y update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient
openstack undercloud upgrade

### UPGRADE OVERCLOUD ###

source stackrc

# workaround to avoid permission issues - this shouldn't be necessary anymore
# sudo chown :stack /var/run/docker.sock

# download container images
openstack overcloud container image upload --verbose --config-file /usr/share/tripleo-common/container-images/overcloud_containers.yaml

# create an envrionment file to make overcloud fetch the images from the undercloud
# (192.168.24.1 is undercloud IP that must be pingable from the overcloud)
echo > ~/containers-default-parameters.yaml 'parameter_defaults:
  DockerNamespace: 192.168.24.1:8787/tripleoupstream
  DockerNamespaceIsRegistry: true
'

# Upgrade overcloud
 openstack overcloud deploy \
    --libvirt-type qemu \
    --ntp-server clock.redhat.com \
    --control-scale 1 \
    --control-flavor oooq_control \
    --compute-flavor oooq_compute \
    --templates $THT \
    -e $THT/environments/low-memory-usage.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
    -e /home/stack/network-environment.yaml \
    -e $THT/environments/docker.yaml \
    -e $THT/environments/major-upgrade-composable-steps-docker.yaml \
-e ~/containers-default-parameters.yaml

Expected result
===============
overcloud was upgraded

Actual result
=============
stack update failed

Environment
===========
1. Ocata

2. default

3. default

Logs & Configs
==============
openstack stack failures list --long overcloud - http://pastebin.test.redhat.com/491167
systemctl from controller node http://pastebin.test.redhat.com/491169

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/471421

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/471421
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=00c21a9a19baeff6718a90a6ed775d060868b395
Submitter: Jenkins
Branch: master

commit 00c21a9a19baeff6718a90a6ed775d060868b395
Author: Pradeep Kilambi <email address hidden>
Date: Tue Jun 6 12:39:47 2017 -0400

    Fix upgrade tasks to use correct service name

    Change-Id: I149ca7cdd939ed7c1767a416bb9569ada163e820
    Closes-bug: #1696089

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b2 development milestone.