external tasks for update (and maybe for everything) are not working anymore.

[Sofer Athlan-Guyot]

In rocky, we updated the image for the overcloud running:

   openstack overcloud external-update run --debug --tags container_image_prepare

In stein that command has no impact:

PLAY [External update steps] ***************************************************

PLAY [External deploy steps] ***************************************************

PLAY RECAP *********************************************************************
compute-0 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
compute-1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
controller-0 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
controller-1 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
controller-2 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
undercloud : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

Thursday 08 August 2019 10:24:21 -0400 (0:00:00.551) 0:00:05.981 *******
===============================================================================

external_update_steps is empty so no worries there, but external_deploy_steps_tasks.yaml isn't and contains the container_image_prepare tag.

But it's not run.

doing:

sudo podman exec -it mistral_executor bash

and adding external to the list of tags to the run command, giving that:

ansible-playbook-3 -vvv /var/lib/mistral/e2e16aca-291b-49a8-bfc1-2a8a1f2f4e40/external_update_steps_playbook.yaml --limit all --module-path /usr/share/ansible-modules --become --become-user root --inventory-file /var/lib/mistral/e2e16aca-291b-49a8-bfc1-2a8a1f2f4e40/inventory.yaml --tags external,container_image_prepare "$@"

in /var/lib/mistral/config-download-latest/ansible-playbook-command.sh

does trigger a run of the container_image_prepare associated tasks.

Problem is with that change I8b3bf3ba3d7c2cfbe1187218c51f619e65efe0e5

When we switched from include to include_tasks we change the tags behavior.

Here's an example:

(undercloud) [stack@undercloud-0 ~]$ cat test.yaml
---
- hosts: all
  gather_facts: no
  tasks:

  - name: "Include tasks file"
    include: tasks.yml

  - name: "Include_tasks tasks file"
    include_tasks: tasks1.yml
(undercloud) [stack@undercloud-0 ~]$ cat tasks.yml
- name: "Debug tag1"
  debug:
    msg: "init"
  tags:
  - tag1
(undercloud) [stack@undercloud-0 ~]$ cat tasks1.yml
- name: "Debug tag1"
  debug:
    msg: "init1"
  tags:
  - tag1

now running:

(undercloud) [stack@undercloud-0 ~]$ ansible-playbook -i my-inventory.yml ./test.yaml -t tag1

PLAY [all] **********************************************************************************************************************************************************************************************************

TASK [Debug tag1] ***************************************************************************************************************************************************************************************************
ok: [undercloud] => {
    "msg": "init"
}
ok: [controller-0] => {
    "msg": "init"
}
ok: [controller-1] => {
    "msg": "init"
}
ok: [controller-2] => {
    "msg": "init"
}
ok: [compute-0] => {
    "msg": "init"
}
ok: [compute-1] => {
    "msg": "init"
}

PLAY RECAP **********************************************************************************************************************************************************************************************************
compute-0 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
compute-1 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
controller-0 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
controller-1 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
controller-2 : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
undercloud : ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

we can see that only the "include" tasks are really included. The include_tasks has a different behavior, it doesn't look into the tasks for the tags there.

See there[1] for more.

[1] https://github.com/ansible/ansible/issues/32015

Adding high, maybe critical, as it should still be broken on master for *all* external commands, but maybe I missed a patch?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/675415

[Sofer Athlan-Guyot (sofer-athlan-guyot)]

Hi,

so deployment tasks are fine because they got converted to "import_tasks" which statically import the tasks and end up with the same behavior than include regarding tags.

upgrade/update tasks broke because "import_tasks" cannot be used in loop, so they got the "include_tasks" which is dynamic but then read its own tag first and broke the old idiom.

Adding the "always" tag seems to be the only solution in that case currently[1]

[1] https://github.com/ansible/ansible/issues/30882#issuecomment-380596557

[OpenStack Infra (hudson-openstack)]

Change abandoned by Athlan-Guyot sofer (<email address hidden>) on branch: master
Review: https://review.opendev.org/675415
Reason: In favor of https://review.opendev.org/#/c/675534

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/675534
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6c675af9babf475cbc825b437b37ff06de4b461b
Submitter: Zuul
Branch: master

commit 6c675af9babf475cbc825b437b37ff06de4b461b
Author: Jose Luis Franco Arza <email address hidden>
Date: Fri Aug 9 08:37:48 2019 +0200

    Add tags always into external update tasks.

    After bumping the Ansible version in Tripleo to 2.6, it was needed
    to perform a change in the include module into include_tasks or
    import_tasks, as include was getting deprecated [0]. The external
    update/upgrades tasks got impacted by that change, but as they use
    a loop to execute the tasks we couldn't use import_tasks.
    The way include_tasks handles the tasks execution depending on the
    tags differs from import_tasks (dynamic vs static) and as a consequence
    when running the external upgrade run passing --tags container_image_prepare
    we didn't see any tasks running. This behavior got fix in [1], which
    seems to be the right way to preserve the import_tasks tags handling as
    explained in [2]. But the external update tasks were missed to patch.

    This patch includes the tags: always statement inside the external
    update tasks and also syncs the variables content we pass into external_update_tasks
    as many of those variables, which we do pass for the external_upgrade_playbook,
    were missing in the external_update_playbook.

    [0] - https://review.opendev.org/#/c/579844/
    [1] - https://review.opendev.org/#/c/639642/1
    [2] - https://github.com/ansible/ansible/issues/30882#issuecomment-380596557
    Closes-Bug: #1839520
    Change-Id: If7e7b4bbb3cead0887384cc543ce37e9ee5396ab

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/677786

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/677786
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=54051f5ecc172a23970863b4bea680d9e86ce5c5
Submitter: Zuul
Branch: stable/stein

commit 54051f5ecc172a23970863b4bea680d9e86ce5c5
Author: Jose Luis Franco Arza <email address hidden>
Date: Fri Aug 9 08:37:48 2019 +0200

    Add tags always into external update tasks.

    After bumping the Ansible version in Tripleo to 2.6, it was needed
    to perform a change in the include module into include_tasks or
    import_tasks, as include was getting deprecated [0]. The external
    update/upgrades tasks got impacted by that change, but as they use
    a loop to execute the tasks we couldn't use import_tasks.
    The way include_tasks handles the tasks execution depending on the
    tags differs from import_tasks (dynamic vs static) and as a consequence
    when running the external upgrade run passing --tags container_image_prepare
    we didn't see any tasks running. This behavior got fix in [1], which
    seems to be the right way to preserve the import_tasks tags handling as
    explained in [2]. But the external update tasks were missed to patch.

    This patch includes the tags: always statement inside the external
    update tasks and also syncs the variables content we pass into external_update_tasks
    as many of those variables, which we do pass for the external_upgrade_playbook,
    were missing in the external_update_playbook.

    [0] - https://review.opendev.org/#/c/579844/
    [1] - https://review.opendev.org/#/c/639642/1
    [2] - https://github.com/ansible/ansible/issues/30882#issuecomment-380596557
    Closes-Bug: #1839520
    Change-Id: If7e7b4bbb3cead0887384cc543ce37e9ee5396ab
    (cherry picked from commit 6c675af9babf475cbc825b437b37ff06de4b461b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.2.0 release.