overcloud upgrade from Ocata to Pike with containerized services failed due to "Could not find the requested service openstack-ceilometer-expirer: cannot disable"

[Artem Hrechanychenko]

Description
===========
I tried to upgrade Ocata cluster to Pike with containerized services.
overcloud upgrade failed:

2017-05-23 14:56:02Z [overcloud-AllNodesDeploySteps-hn3uqk2fsdaj.ControllerUpgrade_Step1.0]: SIGNAL_IN_PROGRESS Signal: deployment 9ba140e0-6ae9-47dc-b2be-815ebbb2eeca failed (2)
2017-05-23 14:56:02Z [overcloud-AllNodesDeploySteps-hn3uqk2fsdaj.ControllerUpgrade_Step1.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2017-05-23 14:56:02Z [overcloud-AllNodesDeploySteps-hn3uqk2fsdaj.ControllerUpgrade_Step1]: CREATE_FAILED Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2017-05-23 14:56:02Z [overcloud-AllNodesDeploySteps-hn3uqk2fsdaj.ControllerUpgrade_Step1]: CREATE_FAILED Error: resources.ControllerUpgrade_Step1.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2017-05-23 14:56:02Z [overcloud-AllNodesDeploySteps-hn3uqk2fsdaj]: UPDATE_FAILED Error: resources.ControllerUpgrade_Step1.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2017-05-23 14:56:02Z [AllNodesDeploySteps]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerUpgrade_Step1.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2017-05-23 14:56:02Z [overcloud]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerUpgrade_Step1.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2

 Stack overcloud UPDATE_FAILED

overcloud.AllNodesDeploySteps.ControllerUpgrade_Step1.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: 9ba140e0-6ae9-47dc-b2be-815ebbb2eeca
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
    TASK [Stop snmp service] *******************************************************
    changed: [localhost]

    TASK [Stop and disable ceilometer_expirer service on upgrade] ******************
    fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Could not find the requested service openstack-ceilometer-expirer: cannot disable"}
     to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/c6a3a135-f38e-4250-bc07-3f184941fc1a_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=17 changed=15 unreachable=0 failed=1

    (truncated, view all with --long)
  deploy_stderr: |

Heat Stack update failed.
Heat Stack update failed.

Steps to reproduce
==================
1)Install undercloud v.Ocata using tripleo-quickstart:
bash quickstart.sh --release ocata $HOST

2)deploy overcloud:
openstack overcloud deploy \
    --libvirt-type qemu \
    --ntp-server clock.redhat.com \
    --control-scale 1 \
    --templates $THT \
    -e $THT/environments/low-memory-usage.yaml \

3) perform upgrade procedure for undercloud
### UPGRADE UNDERCLOUD ###

# master repos
sudo curl -L -o /etc/yum.repos.d/delorean.repo https://trunk.rdoproject.org/centos7-master/current-passed-ci/delorean.repo

sudo curl -L -o /etc/yum.repos.d/delorean-current.repo https://trunk.rdoproject.org/centos7/current/delorean.repo
sudo sed -i 's/\[delorean\]/\[delorean-current\]/' /etc/yum.repos.d/delorean-current.repo
sudo /bin/bash -c "cat <<EOF>>/etc/yum.repos.d/delorean-current.repo

includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common*,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,puppet-*
EOF"
sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo https://trunk.rdoproject.org/centos7/delorean-deps.repo

sudo systemctl stop openstack-*
sudo systemctl stop neutron-*
sudo systemctl stop httpd
sudo yum -y update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient
openstack undercloud upgrade

### UPGRADE OVERCLOUD ###

source stackrc

# workaround to avoid permission issues - this shouldn't be necessary anymore
# sudo chown :stack /var/run/docker.sock

# download container images
openstack overcloud container image upload --verbose --config-file /usr/share/tripleo-common/container-images/overcloud_containers.yaml

# create an envrionment file to make overcloud fetch the images from the undercloud
# (192.168.24.1 is undercloud IP that must be pingable from the overcloud)
echo > ~/containers-default-parameters.yaml 'parameter_defaults:
  DockerNamespace: 192.168.24.1:8787/tripleoupstream
  DockerNamespaceIsRegistry: true
'

# migrate to containers
openstack overcloud deploy \
    --libvirt-type qemu \
    --ntp-server clock.redhat.com \
    --control-scale 1 \
    --templates $THT \
    -e $THT/environments/low-memory-usage.yaml \
    -e $THT/environments/docker.yaml \
    -e $THT/environments/major-upgrade-composable-steps-docker.yaml \
    -e ~/containers-default-parameters.yaml

Expected result
===============
overcloud was upgraded

Actual result
=============
stack update failed

Environment
===========
1. Ocata

2. default

3. default

Logs & Configs
==============
openstack stack failures list --long overcloud - http://pastebin.test.redhat.com/486962
systemctl from controller node http://pastebin.test.redhat.com/486967

[Emilien Macchi (emilienm)]

I'll be fixed by: https://review.openstack.org/#/c/466096/

[Jiří Stránský (jistr)]

The patch merged.