CentOS8-stream: nodepool image fails when starting iptables

[Ronelle Landy]

Using the upstream-centos-8-stream nodepool label to create a node (running on https://review.rdoproject.org/zuul/status - vexxhost), the persistent-firewall zuul-jobs pre task fails to start the iptables service after iptables-services is installed:

Failing task:

https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/persistent-firewall/tasks/persist/RedHat.yaml#L22

Failure trace:

2021-01-08 15:44:01.671509 | TASK [persistent-firewall : Ensure iptables is started]
2021-01-08 15:44:03.096053 | primary | ERROR
2021-01-08 15:44:03.096469 | primary | {
2021-01-08 15:44:03.096554 | primary | "msg": "Could not find the requested service iptables: host"
2021-01-08 15:44:03.096619 | primary | }
2021-01-08 15:44:03.125330 |

Full log:

https://logserver.rdoproject.org/65/31165/16/check/periodic-tripleo-ci-centos-8-standalone-centos8stream-victoria/8f35ec1/job-output.txt

Nodeset added:

https://review.rdoproject.org/r/#/c/31500/2/zuul.d/nodesets.yaml

[Ronelle Landy (rlandy)]

Marking this as a promotion-blocker even though it is not blocking any lines atm - as we should be using the stream node for the stream dependency jobs.

[Javier Peña (jpena-c)]

I think we are hitting https://bugzilla.redhat.com/show_bug.cgi?id=1901449. Ansible 2.9.16 seems to have a workaround, could it be an option?

[Ronelle Landy (rlandy)]

jpena:
the version of ansible is the version on the node at this point.
Since we build the node, can we not install Ansible 2.9.16 on the node itself for stream?

https://logserver.rdoproject.org/41/766541/2/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001/a41174f/logs/undercloud/var/log/extra/package-list-installed.txt.gz shows that ansible.noarch 2.9.16-1.el8 @delorean-ussuri-testing

already gets installed when tripleo code testing begins

same is true on train jobs:

https://logserver.rdoproject.org/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-octavia-train/01db8fa/logs/undercloud/var/log/extra/package-list-installed.txt.gz

[Ronelle Landy (rlandy)]

Switch default ansible-version to 2.9 for zuul

https://review.rdoproject.org/r/#/c/31528/

<jpena_onduty> rlandy: right, but that's not enough yet. I'm trying to get 2.9.16 packaged for software factory

[Ronelle Landy (rlandy)]

https://review.rdoproject.org/r/#/c/31500/ review to make dependency jobs run on stream nodes ...

tested at: https://review.rdoproject.org/r/#/c/31165/ ...

still fails with 2021-01-18 19:11:00.798552 | TASK [persistent-firewall : Ensure iptables is started]
2021-01-18 19:11:02.209768 | primary | ERROR
2021-01-18 19:11:02.210030 | primary | {
2021-01-18 19:11:02.210071 | primary | "msg": "Could not find the requested service iptables: host"

https://review.rdoproject.org/r/#/c/31528/ is not yet merged - so this is expected.

[Ronelle Landy (rlandy)]

works now with ansible 2.9 specified in the job itself:
https://review.rdoproject.org/r/#/c/31500/