[xena] nftables periodic jobs fails with RETRY_LIMIT

[yatin]

Since 18th Nov nftables jobs failing with RETRY_LIMIT and no logs are available. These jobs quickly fails and last task seen running in console is "Preparing job workspace".

This looks a regression with zuul change https://review.opendev.org/c/zuul/zuul/+/900489.

@fungi checked executor logs and it fails as:-
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: [e: a4e70e9b208f479e98334c09305d2013] [build: 3c41d75624274d2e8d7fd62ae332c31d] Exception while executing job
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: Traceback (most recent call last):
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/site-packages/git/util.py", line 1125, in __getitem__
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: return getattr(self, index)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: ^^^^^^^^^^^^^^^^^^^^
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/site-packages/git/util.py", line 1114, in __getattr__
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: return list.__getattribute__(self, attr)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: AttributeError: 'IterableList' object has no attribute '2.3.0'
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: The above exception was the direct cause of the following exception:
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob:
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: Traceback (most recent call last):
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 1185, in do_execute
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: self._execute()
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 1556, in _execute
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: self.preparePlaybooks(args)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2120, in preparePlaybooks
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: self.preparePlaybook(jobdir_playbook, playbook, args)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2177, in preparePlaybook
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: self.prepareRole(jobdir_playbook, role, args)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2367, in prepareRole
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: self.prepareZuulRole(jobdir_playbook, role, args, role_info)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2423, in prepareZuulRole
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: path = self.checkoutUntrustedProject(project, branch, args)
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/site-packages/zuul/executor/server.py", line 2295, in checkoutUntrustedProject
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: repo.getBranchHead(branch).hexsha
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: ^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/site-packages/zuul/merger/merger.py", line 465, in getBranchHead
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: branch_head = repo.heads[branch]
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: ~~~~~~~~~~^^^^^^^^
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: File "/usr/local/lib/python3.11/site-packages/git/util.py", line 1127, in __getitem__
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: raise IndexError("No item found with id %r" % (self._prefix + index)) from e
2023-11-30 14:08:14,253 ERROR zuul.AnsibleJob: IndexError: No item found with id '2.3.0'

The parent job[1] which does pins neutron-tempest-plugin to 2.3.0 also do pass but not the inherited ones, same can be seen in the test patch[2]

Builds:- https://zuul.openstack.org/builds?job_name=neutron-linuxbridge-tempest-plugin-scenario-nftables&job_name=neutron-ovs-tempest-plugin-scenario-iptables_hybrid-nftables&project=openstack%2Fneutron&branch=stable%2Fxena&skip=0

[1] https://zuul.openstack.org/builds?job_name=neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid-xena
[2] https://review.opendev.org/c/openstack/neutron/+/902296

[Dr. Jens Harbott (j-harbott)]

Should be fixed by https://review.opendev.org/c/zuul/zuul/+/902316, just need to restart zuul with it.

[Jeremy Stanley (fungi)]

This problem should go away tomorrow during our automated zuul updates, since https://review.opendev.org/902316 has merged to address the regression.

[yatin (yatinkarel)]

Thanks folks jobs are back to green now:- https://zuul.openstack.org/builds?job_name=neutron-linuxbridge-tempest-plugin-scenario-nftables&job_name=neutron-ovs-tempest-plugin-scenario-iptables_hybrid-nftables&project=openstack%2Fneutron&branch=stable%2Fxena&skip=0