ceph_ansible_skip_tags is getting ignored during multinode overcloud deploy

[chandan kumar]

https://review.opendev.org/#/c/683126/ on tobiko is adding multinode deployment job using featureset 16
<https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset016.yml> and
https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/ci/environments/scenario001-multinode-containers.yaml is used.

During overcloud deploy, https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d6/683126/35/check/tobiko-tripleo-multinode-containers/6d69a07/logs/undercloud/home/zuul/overcloud_deploy.log

TASK [ceph-container-engine : include specific variables] **********************'
2020-03-20 17:59:25 | - 'Friday 20 March 2020 17:58:28 +0000 (0:00:00.208) 0:00:50.451 ********** '
2020-03-20 17:59:25 | - 'ok: [centos-8-vexxhost-sjc1-0015363305] => (item=/usr/share/ceph-ansible/roles/ceph-container-engine/vars/RedHat.yml) => changed=false '
2020-03-20 17:59:25 | - ' container_binding_name: python-docker-py'
2020-03-20 17:59:25 | - ' container_package_name: docker'
2020-03-20 17:59:25 | - ' container_service_name: docker'
2020-03-20 17:59:25 | - ' ansible_included_var_files:'
2020-03-20 17:59:25 | - ' - /usr/share/ceph-ansible/roles/ceph-container-engine/vars/RedHat.yml'
2020-03-20 17:59:25 | - ' item: /usr/share/ceph-ansible/roles/ceph-container-engine/vars/RedHat.yml'
2020-03-20 17:59:25 | - 'TASK [ceph-container-engine : debian based systems tasks] **********************'
2020-03-20 17:59:25 | - 'Friday 20 March 2020 17:58:28 +0000 (0:00:00.208) 0:00:50.660 ********** '
2020-03-20 17:59:25 | - 'TASK [ceph-container-engine : enable extras on centos] *************************'
2020-03-20 17:59:25 | - 'Friday 20 March 2020 17:58:28 +0000 (0:00:00.167) 0:00:50.827 ********** '
2020-03-20 17:59:25 | - 'TASK [ceph-container-engine : install container and lvm2 packages] *************'
2020-03-20 17:59:25 | - 'Friday 20 March 2020 17:58:28 +0000 (0:00:00.096) 0:00:50.924 ********** '
2020-03-20 17:59:25 | - 'FAILED - RETRYING: install container and lvm2 packages (3 retries left).'
2020-03-20 17:59:25 | - 'FAILED - RETRYING: install container and lvm2 packages (2 retries left).'
2020-03-20 17:59:25 | - 'FAILED - RETRYING: install container and lvm2 packages (1 retries left).'
2020-03-20 17:59:25 | - 'fatal: [centos-8-vexxhost-sjc1-0015363305]: FAILED! => changed=false '
2020-03-20 17:59:25 | - ' attempts: 3'
2020-03-20 17:59:25 | - ' failures:'
2020-03-20 17:59:25 | - ' - No package docker available.'
2020-03-20 17:59:25 | - ' - No package python-docker-py available.'
2020-03-20 17:59:25 | - ' msg: Failed to install some of the specified packages'
2020-03-20 17:59:25 | - ' rc: 1'
2020-03-20 17:59:25 | - ' results: []'
2020-03-20 17:59:25 | - NO MORE HOSTS LEFT *************************************************************

It is trying to install lvm packages but by default in tripleo-ansible repo
https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_ceph_common/defaults/main.yml#L64
ceph_ansible_skip_tags: 'package-install,with_pkg' which will skip the package installation but package installation is happening there.

While looking at https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d6/683126/35/check/tobiko-tripleo-multinode-containers/6d69a07/logs/undercloud/var/lib/mistral/config-download-latest/ceph-ansible/ceph_ansible_command.sh

#!/usr/bin/env bash
set -e
echo "Running $0" >> /var/lib/mistral/overcloud/ceph-ansible/ceph_ansible_command.log
ANSIBLE_ACTION_PLUGINS=/usr/share/ceph-ansible/plugins/actions/ ANSIBLE_CALLBACK_PLUGINS=/usr/share/ceph-ansible/plugins/callback/ ANSIBLE_FILTER_PLUGINS=/usr/share/ceph-ansible/plugins/filter/ ANSIBLE_ROLES_PATH=/usr/share/ceph-ansible/roles/ ANSIBLE_LOG_PATH="/var/lib/mistral/overcloud/ceph-ansible/ceph_ansible_command.log" ANSIBLE_LIBRARY=/usr/share/ceph-ansible/library/ ANSIBLE_CONFIG=/usr/share/ceph-ansible/ansible.cfg ANSIBLE_REMOTE_TEMP=/tmp/ceph_ansible_tmp ANSIBLE_FORKS=25 ANSIBLE_GATHER_TIMEOUT=60 ANSIBLE_HOST_KEY_CHECKING=False ANSIBLE_PRIVATE_KEY_FILE=/home/zuul/.ssh/id_rsa_tripleo ANSIBLE_SSH_RETRIES=4 DEFAULT_FORKS=3 ansible-playbook -e ansible_python_interpreter=/usr/bin/python3 -v --extra-vars @/var/lib/mistral/overcloud/ceph-ansible/extra_vars.yml -i /var/lib/mistral/overcloud/ceph-ansible/inventory.yml /usr/share/ceph-ansible/site-container.yml.sample 2>&1

Above tag ceph_ansible_skip_tags: 'package-install,with_pkg' is not there, leading to installation of packages like lvm on overcloud.
There might be something wrong with ceph_ansible_skip_tags and it needs to be investigated.

[John Fulton (jfulton-org)]

tripleo-ci-centos-8-scenario001-standalone + tripleo-ci-centos-8-scenario004-standalone both are not affected by this bug as I see:

 --skip-tags package-install,with_pkg in the generated ceph_ansible_command.sh

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_536/714482/1/check/tripleo-ci-centos-8-scenario001-standalone/5365145/logs/undercloud/home/zuul/standalone-ansible-jg52oo_n/ceph-ansible/ceph_ansible_command.sh

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_960/714482/1/check/tripleo-ci-centos-8-scenario004-standalone/96028f8/logs/undercloud/home/zuul/standalone-ansible-hbrc6ema/ceph-ansible/

[John Fulton (jfulton-org)]

We're passing it in Heat:

https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L80-L83

The problem was that the job definition was overriding it:

https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/ci/environments/scenario001-multinode-containers.yaml#L144

So this can be fixed by updating the job definition.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/714104
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6feca7e129a5ecf4af089c93abb1ec647b87d3dd
Submitter: Zuul
Branch: master

commit 6feca7e129a5ecf4af089c93abb1ec647b87d3dd
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Fri Mar 20 18:21:57 2020 +0530

    Remove CephAnsibleSkipTags override

    We want to skip tags for installing packages as is the default.
    https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_common/defaults/main.yml#L64

    Depends-On: https://review.opendev.org/#/c/714768/

    Closes-Bug: #1868731

    Change-Id: I85af342e3a546e1c195c7ba3d39d53e85e6fdd7e
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.2.0 release.