overcloud upgrade failed: " no crontab for ceilometer"

[Artem Hrechanychenko]

Description
===========
overcloud.AllNodesDeploySteps.ControllerUpgrade_Step2.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: a93e5e5c-f1c3-4caa-90a6-21913ebde429
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |

    PLAY [localhost] ***************************************************************

    TASK [Gathering Facts] *********************************************************
    ok: [localhost]

    TASK [Stop and disable aodh service (running under httpd)] *********************
    changed: [localhost]

    TASK [Stop and disable openstack-aodh-evaluator service] ***********************
    changed: [localhost]

    TASK [Stop and disable openstack-aodh-listener service] ************************
    changed: [localhost]

    TASK [Stop and disable openstack-aodh-notifier service] ************************
    changed: [localhost]

    TASK [Stop and disable ceilometer agent central service] ***********************
    changed: [localhost]

    TASK [Stop and disable ceilometer agent notification service] ******************
    changed: [localhost]

    TASK [Stop and disable cinder_api service] *************************************
    ok: [localhost]

    TASK [Stop and disable cinder_scheduler service] *******************************
    changed: [localhost]

    TASK [Stop and disable glance_api service] *************************************
    changed: [localhost]

    TASK [Stop and disable httpd service] ******************************************
    ok: [localhost]

    TASK [Stop and disable openstack-gnocchi-metricd service] **********************
    changed: [localhost]

    TASK [Stop and disable openstack-gnocchi-statsd service] ***********************
    changed: [localhost]

    TASK [Stop and disable heat_api service] ***************************************
    ok: [localhost]

    TASK [Stop and disable heat_api_cfn service] ***********************************
    ok: [localhost]

    TASK [Check if heat_api_cloudwatch is deployed] ********************************
    fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["systemctl", "is-enabled", "openstack-heat-api-cloudwatch"], "delta": "0:00:00.004891", "end": "2017-06-29 09:10:45.342845", "failed": true, "rc": 1, "start": "2017-06-29 09:10:45.337954", "stderr": "", "stderr_lines": [], "stdout": "disabled", "stdout_lines": ["disabled"]}
    ...ignoring

    TASK [Stop and disable heat_engine service] ************************************
    changed: [localhost]

    TASK [Stop and disable horizon service (running under httpd)] ******************
    ok: [localhost]

    TASK [Stop and disable iscsid service] *****************************************
    fatal: [localhost]: FAILED! => {"failed": true, "msg": "The conditional check 'stat_iscsid_service.stat.exists' failed. The error was: error while evaluating conditional (stat_iscsid_service.stat.exists): 'stat_iscsid_service' is undefined"}
     to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/6d340431-7033-4c95-9318-716da99aae0a_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=18 changed=12 unreachable=0 failed=1

  deploy_stderr: |

overcloud.AllNodesDeploySteps.ControllerUpgrade_Step1.0:
  resource_type: OS::Heat::SoftwareDeployment
  physical_resource_id: eef76217-214a-466d-ad86-6e979a34de14
  status: UPDATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |

    PLAY [localhost] ***************************************************************

    TASK [Gathering Facts] *********************************************************
    ok: [localhost]

    TASK [Remove ceilometer expirer cron tab on upgrade] ***************************
    fatal: [localhost]: FAILED! => {"changed": true, "cmd": "/usr/bin/crontab -u ceilometer -r", "delta": "0:00:00.006122", "end": "2017-06-29 10:42:28.228248", "failed": true, "rc": 1, "start": "2017-06-29 10:42:28.222126", "stderr": "no crontab for ceilometer", "stderr_lines": ["no crontab for ceilometer"], "stdout": "", "stdout_lines": []}
     to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/db693d1b-cc4f-4b63-b06e-4d433d97462d_playbook.retry

    PLAY RECAP *********************************************************************
    localhost : ok=1 changed=0 unreachable=0 failed=1

  deploy_stderr: |
Steps to reproduce
==================
upgrade ocata to pike(regarding to update OSP11 to OSP11 latest+RHEL7.4; upgrade OSP11->OSP12 undercloud,overcloud)

infrared virsh -v --host-address 10.9.76.22 --host-key ~/.ssh/id_rsa --cleanup yes && infrared virsh -v --host-address 10.9.76.22 --host-key ~/.ssh/id_rsa --topology-nodes undercloud:1,controller:1,compute:1 -e override.controller.cpu=8 -e override.controller.memory=16384 -e override.undercloud.disks.disk1.size=100G && infrared tripleo-undercloud --version 11 --images-task=rpm && infrared tripleo-overcloud -v --introspect yes --tagging yes --post no --deployment-files virt --version 11 --deploy yes

2.Upgrade undercloud and overcloud to osp11 latest+ rhel7.4
ir tripleo-undercloud -v --update-undercloud yes --mirror qeos --build 7.4-testing --osrelease 7.4

ir tripleo-overcloud -v --updateto 7.4-testing --deployment-files virt --mirror qeos --osrelease 7.4
#Please note, during yum update stage on overcloud controller run "sudo pcs cluster start and sudo pcs cleanup resources" - w/a for https://bugzilla.redhat.com/show_bug.cgi?id=1464588

3. Upgrade undercloud and overcloud to osp12
http://etherpad.corp.redhat.com/osp12-upgrade
#please note, before updating overcloud apply w/a for
https://bugzilla.redhat.com/show_bug.cgi?id=1460421
https://bugzilla.redhat.com/show_bug.cgi?id=1463227
https://bugzilla.redhat.com/show_bug.cgi?id=1466219

Expected result
===============
overcloud upgrade failed

Actual result
=============
overcloud upgrade was succeed

Logs & Configs
==============

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/478909

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/478909
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=af5eaf11337c363bdf9a126744ceddb94a80343e
Submitter: Jenkins
Branch: master

commit af5eaf11337c363bdf9a126744ceddb94a80343e
Author: Jiri Stransky <email address hidden>
Date: Thu Jun 29 14:25:21 2017 +0200

    Make ceilometer crontab removal idempotent

    When we re-execute an upgrade and the crontab has already been
    removed, the crontab removal returns 1, saying "no crontab for
    ceilometer", and the upgrade fails. This change makes the removal
    idempotent.

    Change-Id: Ic955fb67bb2f7afde44291f7db3293c88f167566
    Closes-Bug: #1701250

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b3 development milestone.