Wheels build may not work properly on multi-distro environment

[Damian Dąbrowski]

* Problem description:

Recently we solved one issue related to wheels build process in multi-distro environments: https://opendev.org/openstack/ansible-role-python_venv_build/commit/28f89b80a8fbac5018638d512978eec210269649
Unfortunately I found another one, and this time it's much harder to fix it.

I think it's the best to explain this issue on certain example.

Let's imagine, You are running ubuntu 18.04 and want to upgrade to 20.04.
You have 3 shared-infra nodes(controllers). They exist in ansible inventory in a following order: control02, control01, control03
control02 is running lsyncd on repo containers so based on the documentation, it shouldn't be upgraded first, so You upgrade control03 as a first host.

So You have re-provisioned control03 with a new operating system, now it looks like this:
- control02(lsyncd master, ubuntu 18.04)
- control01(ubuntu 18.04)
- control03(ubuntu 20.04)

And You want to install openstack services on control03.
Keystone can't be run with limits, so You need to run it against all hosts. It runs in a following serial: "{{ keystone_serial | default(['1', '100%']) }}"

So keystone playbook runs in two batches:
1.control02
first host, nothing really changes as wheels are already built for ubuntu 18.04

2. control01(18.04), control03(20.04)
And there is our main issue. python_venv_wheel_build.yml is marked with run_once, so it may run only against control01.
In this case wheels for ubuntu 20.04 won't be built at all.

https://opendev.org/openstack/ansible-role-python_venv_build/src/commit/3dddb491a999e8a8a1409ef98a2aed84f419c7df/tasks/main.yml#L67

* Proposed solution:

TBH I'm not sure how we can fix it. We can't just drop run_once because it will trigger multiple wheel builds, especially when combined with venv_rebuild=true.
It will drastically slow down the process and may also DDoS gitea(https://opendev.org/openstack/ansible-role-python_venv_build/commit/de731e7f9e15a99e9587b42de175531bab3f401e)

Maybe it's possible to replace run_once with something limiting wheels build process only to first host of each distribution. But from top of my head, I don't know how we can achieve it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/848106

[Dmitriy Rabotyagov (noonedeadpunk)]

I think one possible workaround would be to limit play against each group of host with different architecture and/or OS distribution to make wheels build for each group.
But that needs to be done with each playbook run, which is not that great or convenient.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/848106
Committed: https://opendev.org/openstack/ansible-role-python_venv_build/commit/57a2f226ebca7a2ec21920d4fd2b7ccf45490684
Submitter: "Zuul (22348)"
Branch: master

commit 57a2f226ebca7a2ec21920d4fd2b7ccf45490684
Author: Dmitriy Rabotyagov <email address hidden>
Date: Wed Jun 29 15:08:30 2022 +0200

    Run wheels build for each unique distro/arch

    There may be multiple architecures and OS versions in
    ansible_play_hosts. However, with run_once we build wheels only
    for one selected distro and do not repsect multi-arch/multi-distro
    setups.
    Instead of run_once we need to select single (and first in play)
    host of each architecture and distro and delegate wheels building
    from it. That is needed, because venv_build_host is selected based
    on the facts gathered for current inventory_hostname and will
    depend on it's arch/distro.

    Change-Id: I492d17169538ad2768e28f7c48314bdec407ab36
    Closes-Bug: #1964535

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/849004

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/849004
Committed: https://opendev.org/openstack/ansible-role-python_venv_build/commit/8fadaf0278d9c0313ff0d27d5e3182bd9ae3703e
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 8fadaf0278d9c0313ff0d27d5e3182bd9ae3703e
Author: Dmitriy Rabotyagov <email address hidden>
Date: Wed Jun 29 15:08:30 2022 +0200

    Run wheels build for each unique distro/arch

    There may be multiple architecures and OS versions in
    ansible_play_hosts. However, with run_once we build wheels only
    for one selected distro and do not repsect multi-arch/multi-distro
    setups.
    Instead of run_once we need to select single (and first in play)
    host of each architecture and distro and delegate wheels building
    from it. That is needed, because venv_build_host is selected based
    on the facts gathered for current inventory_hostname and will
    depend on it's arch/distro.

    Change-Id: I492d17169538ad2768e28f7c48314bdec407ab36
    Closes-Bug: #1964535
    (cherry picked from commit 57a2f226ebca7a2ec21920d4fd2b7ccf45490684)