master OVB jobs are failing overcloud deployment - UNREACHABLE | Manage firewall rules

[Ronelle Landy]

periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master is failing at the overcloud deploy step with the following error:

2022-04-14 17:21:29 | 2022-04-14 17:21:29.985003 | fa163e91-3f26-4179-e89c-000000001f2e | UNREACHABLE | Manage firewall rules | overcloud-controller-0
2022-04-14 17:21:29 | 2022-04-14 17:21:29.988904 | fa163e91-3f26-4179-e89c-000000001f2e | TIMING | tripleo_firewall : Manage firewall rules | overcloud-controller-0 | 0:11:58.043640 | 528.96s
2022-04-14 17:21:29 | 2022-04-14 17:21:29.989743 | fa163e91-3f26-4179-e89c-000000001f2e | UNREACHABLE | Manage firewall rules | overcloud-controller-2
2022-04-14 17:21:29 | 2022-04-14 17:21:29.990287 | fa163e91-3f26-4179-e89c-000000001f2e | TIMING | tripleo_firewall : Manage firewall rules | overcloud-controller-2 | 0:11:58.045046 | 528.87s
2022-04-14 17:21:29 | 2022-04-14 17:21:29.990971 | fa163e91-3f26-4179-e89c-000000001f2e | UNREACHABLE | Manage firewall rules | overcloud-controller-1
2022-04-14 17:21:29 | 2022-04-14 17:21:29.991492 | fa163e91-3f26-4179-e89c-000000001f2e | TIMING | tripleo_firewall : Manage firewall rules | overcloud-controller-1 | 0:11:58.046252 | 529.17s
2022-04-14 17:21:29 | 2022-04-14 17:21:29.992079 | fa163e91-3f26-4179-e89c-0000000020c0 | UNREACHABLE | Manage firewall rules | overcloud-novacompute-0
2022-04-14 17:21:29 | 2022-04-14 17:21:29.992643 | fa163e91-3f26-4179-e89c-0000000020c0 | TIMING | tripleo_firewall : Manage firewall rules | overcloud-novacompute-0 | 0:11:58.047403 | 528.95s

The full log is below:
https://logserver.rdoproject.org/cb/cbabb15dac13bde54666606ec978b75cb2e205a0/openstack-periodic-integration-main/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/dbe28f8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

fs035 is showing the same error:
https://logserver.rdoproject.org/cb/cbabb15dac13bde54666606ec978b75cb2e205a0/openstack-periodic-integration-main/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/47e0d98/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

The error started on 04/13.

[Ronelle Landy (rlandy)]

Tengu pointed to possible cause: https://review.opendev.org/c/openstack/tripleo-ansible/+/836871

with fix - that has a -1 atm: https://review.opendev.org/c/openstack/tripleo-ansible/+/837994

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/838097

[Rabi Mishra (rabi)]

https://review.opendev.org/c/openstack/python-tripleoclient/+/836376/4/tripleoclient/utils.py#b471 maybe the reason for it to surface in CI?

[Sandeep Yadav (sandeepyadav93)]

Thank you Rabi,

We are testing revert[1] of the above patch at [2].

[1] https://review.opendev.org/c/openstack/python-tripleoclient/+/838228
[2] https://review.rdoproject.org/r/c/testproject/+/39420

[Sandeep Yadav (sandeepyadav93)]

Thank you Rabi,

Revert [1] worked, We have already passed the deployment step.

I have also proposed a patch[2] to expand ovb coverage for patches against python-tripleoclient.

[1] https://review.opendev.org/c/openstack/python-tripleoclient/+/838228
[2] https://review.rdoproject.org/r/c/config/+/42021

[Sandeep Yadav (sandeepyadav93)]

fs001 Job is back to green[1] after merge of [2]

[1] https://review.rdoproject.org/zuul/build/ca61e547330e4d40baaf9c10036cb77b
[2] https://review.opendev.org/c/openstack/python-tripleoclient/+/838228

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/839163

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Cedric Jeanneret <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/838097
Reason: Replaced by Rhttps://review.opendev.org/c/openstack/tripleo-ansible/+/839163

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Cedric Jeanneret <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-ansible/+/839163
Reason: Since we're moving toward nftables instead of iptables, this work isn't relevant anymore (already implemented in nftables since day-1 of the thinking)