Invalid facts on external upgrade run

[Sergii Golovatiuk]

Analyzing CI failure [1] I have found the following situation.

The job tries to upgrade from rocky (which is docker based) to stein (which is podman based)

Upgrade steps were run executing tripleo-docker-rm, removing docker as part of upgrade to podman.

After upgrade steps "External upgrade" steps where processed. On External upgrade step 1, "Online data migration for Cinder" happened which is described as

     external_upgrade_tasks:
        - when: step|int == 1
          block:
            - name: Online data migration for Cinder
              command: "{{ container_cli }} exec cinder_api cinder-manage db online_data_migrations"
              delegate_to: "{{ groups['cinder_api'][0] }}"
              become: true
              tags:
                - online_upgrade
                - online_upgrade_cinder

So, container_cli should be resolved as podman but it's not as container_cli fact was not updated. Thus, ansible considered that container_cli was still docker.

Steps to reproduce
==================
Deploy standalone rocky
Upgrade to standalone stein

Expected result
===============
Upgraded environment

Actual result
=============
Failed standalone upgrade due to wrong fact.

[1] https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_8c2/752938/1/check/tripleo-ci-centos-7-standalone-upgrade-stein/8c2ea1e/logs/undercloud/home/zuul/standalone_upgrade.log

[Sergii Golovatiuk (sgolovatiuk)]

I guess in tasks/tripleo_docker_purge.yml we should rather do

    - name: clearing facts
      meta: clear_facts

instead of

     - name: Refresh hardware facts
       setup:
         gather_subset:
           - hardware

[Kevin Carter (kevin-carter)]

+1 on to using clear facts. While this would update the facts globally for the playbook context, forcing everything to refresh, it would ensure we have the most up-to-date information when performing this action which I think is worth the "cost" of regathering facts.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/759303

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/759303
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7e937946005191a9362f61f0ab91b317a267bcd6
Submitter: Zuul
Branch: stable/stein

commit 7e937946005191a9362f61f0ab91b317a267bcd6
Author: Kevin Carter <email address hidden>
Date: Thu Oct 22 12:09:25 2020 -0500

    Add clear facts to the start of all major playbooks

    This change will ensure that any and all ansible facts are cleared
    when starting any and all core framework interactions. By clearning
    the cached facts we'll ensure we're not running into legacy or
    otherwise outdated fact information.

    This change is adapted from master [0].

    [0] https://github.com/openstack/tripleo-heat-templates/commit/1f9942fd755a168f8a9b75afa09e8af7027d61d4

    Change-Id: I2410f86db063ef3c037baf2bc716075bda40f11c
    Closes-Bug: 1900836
    Signed-off-by: Kevin Carter <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates stein-eol release.