keystone upgrade to Queens failure due to new handler

[Jean-Philippe Evrard]

A new "Manage LB" handler is now notified in Queens like in [1], but it causes issues in role tests. We probably need to define a dummy or a proper "Manage LB" handler in role tests, else we hit this issue [2].

I haven't analysed the root cause of the handler being undefined in tests yet, but it's probably a simple change in tests repo to apply.

[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/master/tasks/keystone_install.yml#L161

[2]: http://logs.openstack.org/54/524954/25/check/openstack-ansible-upgrade-ubuntu-xenial/f5eb385/job-output.txt.gz#_2018-01-05_09_44_02_522391

[Jean-Philippe Evrard (jean-philippe-evrard)]

I had a look today, but I can't seem to find the notify chain that leads to "Manage LB" or the previous definition of a handler named "Manage LB" in the whole process of a role upgrade.

Need further investigation to find the root cause task.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/532069

[Jean-Philippe Evrard (jean-philippe-evrard)]

This seems to come from ansible 2.4.

As ansible2.4 is technically required to do everything until the os_previous_role, we are installing ansible 2.4.2.0 in the tox environment (see tox.ini fetching the requirements from git.openstack.org, master branch).

We are not reverting ansible to ansible 2.3 for the previous role execution.

Reverting to ansible 2.3 for the upgrade test solves this issue appearing during os_previous_role run.

Which leads me to think that ansible 2.4 is loading handlers and notifications in a different way, causing a wrong definition of the handler during the os_previous_keystone role run.

[Jean-Philippe Evrard (jean-philippe-evrard)]

In ansible 2.4 this task [1] includes the file located in /opt/openstack-ansible-os_keystone/tasks/keystone_key_populate.yml instead of loading the file /root/.ansible/roles/os_previous_keystone/tasks/keystone_key_setup.yml . Now I need to check why and how to fix it.

[1]: https://github.com/openstack/openstack-ansible-os_keystone/blob/stable/pike/tasks/keystone_key_setup.yml#L16

[Jean-Philippe Evrard (jean-philippe-evrard)]

Wrote a minimum test case in https://github.com/evrardjp/ansible-pocs/tree/acd15abbc3ba126fdd6595c845fef3d5dc24a84d Will dig deeper now.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/532470

[Jean-Philippe Evrard (jean-philippe-evrard)]

I did a patch to quickly fix the issue.
This patch is, conforming to the discussion, containing a venv build script to allow us to liberate from tox in the future.

This doesn't fix the ansible 2.4 part yet, which we should also do in the near future.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jean-Philippe Evrard (<email address hidden>) on branch: master
Review: https://review.openstack.org/532069

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/532470
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=91b7524603650edbef984fd9fe83e352b90604d2
Submitter: Zuul
Branch: master

commit 91b7524603650edbef984fd9fe83e352b90604d2
Author: Jean-Philippe Evrard <email address hidden>
Date: Wed Jan 10 09:21:57 2018 +0000

    Use a venv with previous ansible version for upgrades

    When doing the role upgrade, we run every playbook with current's
    branch ansible. This is normal for the infrastructure bits (which
    are running on current branch), but not for the installation of
    the previous branch's role, which should use previous branch's
    environment.

    This commit adds:
    - A new venv build script, containing the previous version of ansible
    - A split between the infrastructure bits (running current ansible),
      the previous role run, and the upgrade of the role.

    Change-Id: I91fe4e867750d7cee9ad7d84b005eb0231849df9
    Signed-off-by: Jean-Philippe Evrard <email address hidden>
    Closes-Bug: #1741471

[Jean-Philippe Evrard (jean-philippe-evrard)]

I submitted a very ugly fix, to get the conversation started: https://github.com/ansible/ansible/pull/34790/commits

Waiting for more eyes to see how it goes.

[Jean-Philippe Evrard (jean-philippe-evrard)]

We should be ok for our process (the patch above merged, which checks out ansible 2.3 before running the previous role), but I still think the precedence issue should get fixed in ansible 2.4 and above (or at least we should introduce tests to avoid regressions). This bug should be kept in our list of things to watch in ansible.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/536372

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/536374

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_keystone 17.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/539875

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/539883

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/539896

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/539897

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/539899

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/539902

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/536372
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-tests/commit/?id=f51a22988de85e44b122af8046eedc396de98032
Submitter: Zuul
Branch: master

commit f51a22988de85e44b122af8046eedc396de98032
Author: Markos Chandras <email address hidden>
Date: Mon Jan 22 13:57:57 2018 +0000

    Use a venv with previous ansible version for upgrades

    When doing the role upgrade, we run every playbook with current's
    branch ansible. This is normal for the infrastructure bits (which
    are running on current branch), but not for the installation of
    the previous branch's role, which should use previous branch's
    environment. This script already exists for some OSA roles to
    facilitate upgrades but we are now moving it to the tests
    repository so all roles can benefit from it.

    (Original patch 91b7524603650edbef984fd9fe83e352b90604d2)
    Link: https://github.com/ansible/ansible/pull/34790
    Co-Authored-By: Jean-Philippe Evrard <email address hidden>
    Closes-Bug: #1741471

    Change-Id: I7ebb045885dd645b820de2b7f75b46c755c015f6

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/536374
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=e4456ed4c1f7e3506e4698e17dc94b4778517bf9
Submitter: Zuul
Branch: master

commit e4456ed4c1f7e3506e4698e17dc94b4778517bf9
Author: Markos Chandras <email address hidden>
Date: Mon Jan 22 14:11:21 2018 +0000

    Use a venv with previous ansible version for upgrades

    When doing the role upgrade, we run every playbook with current's
    branch ansible. This is normal for the infrastructure bits (which
    are running on current branch), but not for the installation of
    the previous branch's role, which should use previous branch's
    environment.

    Related-Bug: #1741471
    Depends-On: I7ebb045885dd645b820de2b7f75b46c755c015f6
    Change-Id: I7bc4dd644c48b403e9e669cb0ea460d9d1149410

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539902
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_glance/commit/?id=594412d43ad5af4a13c510eca6d967e9e231680a
Submitter: Zuul
Branch: master

commit 594412d43ad5af4a13c510eca6d967e9e231680a
Author: Jean-Philippe Evrard <email address hidden>
Date: Thu Feb 1 10:52:13 2018 +0000

    Use a venv with previous ansible version for upgrades

    When doing the role upgrade, we run every playbook with current's
    branch ansible. This is normal for the infrastructure bits (which
    are running on current branch), but not for the installation of
    the previous branch's role, which should use previous branch's
    environment.

    This patch also renames test-upgrade-pre to test-upgrade, to
    be in line with other roles.

    Depends-On: I7ebb045885dd645b820de2b7f75b46c755c015f6
    Change-Id: I3e7198d368ef1d7373bb036f60a75d1ddfa4b45c
    Related-Bug: #1741471

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539896
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=f1144a762c5f89c9ff2e127a355f23a7bfe7000f
Submitter: Zuul
Branch: master

commit f1144a762c5f89c9ff2e127a355f23a7bfe7000f
Author: Jean-Philippe Evrard <email address hidden>
Date: Thu Feb 1 10:38:35 2018 +0000

    Use a venv with previous ansible version for upgrades

    When doing the role upgrade, we run every playbook with current's
    branch ansible. This is normal for the infrastructure bits (which
    are running on current branch), but not for the installation of
    the previous branch's role, which should use previous branch's
    environment.

    Depends-On: I7ebb045885dd645b820de2b7f75b46c755c015f6
    Change-Id: Ied3168a4a2df385f8cba8f89b2efab208e639e55
    Related-Bug: #1741471

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539899
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_swift/commit/?id=45bab4b6acf94475b4b1007fbc27f20fc2ebc792
Submitter: Zuul
Branch: master

commit 45bab4b6acf94475b4b1007fbc27f20fc2ebc792
Author: Jean-Philippe Evrard <email address hidden>
Date: Thu Feb 1 10:45:59 2018 +0000

    Use a venv with previous ansible version for upgrades

    When doing the role upgrade, we run every playbook with current's
    branch ansible. This is normal for the infrastructure bits (which
    are running on current branch), but not for the installation of
    the previous branch's role, which should use previous branch's
    environment.

    Depends-On: I7ebb045885dd645b820de2b7f75b46c755c015f6
    Change-Id: I7069c906395b37e8af20db82d88d8a069faf5f79
    Related-Bug: #1741471

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539875
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=113aa975b18172a57ad60e519651d9c209e8639f
Submitter: Zuul
Branch: master

commit 113aa975b18172a57ad60e519651d9c209e8639f
Author: Jean-Philippe Evrard <email address hidden>
Date: Thu Feb 1 09:50:37 2018 +0000

    Use a venv with previous ansible version for upgrades

    When doing the role upgrade, we run every playbook with current's
    branch ansible. This is normal for the infrastructure bits (which
    are running on current branch), but not for the installation of
    the previous branch's role, which should use previous branch's
    environment.

    Change-Id: I88e2616db193487905e1e1552695ec4847bcb4f9
    Depends-On: I7ebb045885dd645b820de2b7f75b46c755c015f6
    Related-Bug: #1741471

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539897
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_neutron/commit/?id=3a47ee875de886623510f90fcee24425472d4a07
Submitter: Zuul
Branch: master

commit 3a47ee875de886623510f90fcee24425472d4a07
Author: Jean-Philippe Evrard <email address hidden>
Date: Thu Feb 1 10:42:18 2018 +0000

    Use a venv with previous ansible version for upgrades

    When doing the role upgrade, we run every playbook with current's
    branch ansible. This is normal for the infrastructure bits (which
    are running on current branch), but not for the installation of
    the previous branch's role, which should use previous branch's
    environment.

    Depends-On: I7ebb045885dd645b820de2b7f75b46c755c015f6
    Change-Id: I5fbf34688b8930d47cbb36866fe2436ce143ee00
    Related-Bug: #1741471

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539883
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_keystone/commit/?id=f00d412e62424cce91ebf91451bf5e3d4c26c66b
Submitter: Zuul
Branch: master

commit f00d412e62424cce91ebf91451bf5e3d4c26c66b
Author: Jean-Philippe Evrard <email address hidden>
Date: Thu Feb 1 10:10:36 2018 +0000

    Use the tests ansible-previous venv

    When doing the role upgrade, we run every playbook with current's
    branch ansible. This is normal for the infrastructure bits (which
    are running on current branch), but not for the installation of
    the previous branch's role, which should use previous branch's
    environment.

    The tests repo now has this venv, so we should make use of it.

    Depends-On: I7ebb045885dd645b820de2b7f75b46c755c015f6
    Change-Id: I579a3f7b641e02f40ed1b21f523aecbb9f16835a
    Related-Bug: #1741471

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/543236

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/543237

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/543236
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-tests/commit/?id=b62a1351fa8b23e42205a0725ee9eaeb8426d429
Submitter: Zuul
Branch: stable/pike

commit b62a1351fa8b23e42205a0725ee9eaeb8426d429
Author: Markos Chandras <email address hidden>
Date: Mon Jan 22 13:57:57 2018 +0000

    Use a venv with previous ansible version for upgrades

    When doing the role upgrade, we run every playbook with current's
    branch ansible. This is normal for the infrastructure bits (which
    are running on current branch), but not for the installation of
    the previous branch's role, which should use previous branch's
    environment. This script already exists for some OSA roles to
    facilitate upgrades but we are now moving it to the tests
    repository so all roles can benefit from it.

    (Original patch 91b7524603650edbef984fd9fe83e352b90604d2)
    Link: https://github.com/ansible/ansible/pull/34790
    Co-Authored-By: Jean-Philippe Evrard <email address hidden>
    Closes-Bug: #1741471

    Change-Id: I7ebb045885dd645b820de2b7f75b46c755c015f6
    (cherry picked from commit f51a22988de85e44b122af8046eedc396de98032)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/543237
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-galera_server/commit/?id=016eaa178f164d80b350aa53e1c031eb14870357
Submitter: Zuul
Branch: stable/pike

commit 016eaa178f164d80b350aa53e1c031eb14870357
Author: Markos Chandras <email address hidden>
Date: Mon Jan 22 14:11:21 2018 +0000

    Use a venv with previous ansible version for upgrades

    When doing the role upgrade, we run every playbook with current's
    branch ansible. This is normal for the infrastructure bits (which
    are running on current branch), but not for the installation of
    the previous branch's role, which should use previous branch's
    environment.

    Related-Bug: #1741471
    Depends-On: I7ebb045885dd645b820de2b7f75b46c755c015f6
    Change-Id: I7bc4dd644c48b403e9e669cb0ea460d9d1149410
    (cherry picked from commit e4456ed4c1f7e3506e4698e17dc94b4778517bf9)