Tempest test TestNetworkBasicOps.test_port_security_macspoofing_port sometimes is failing due to ping issue

[Slawek Kaplonski]

It happened at least twice recently in the neutron-ovn-tempest-slow job:

    https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_992/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-slow/992d108/testr_results.html

    https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_64e/periodic/opendev.org/openstack/neutron/master/neutron-ovn-tempest-slow/64eb479/testr_results.html

Stacktrace:

Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 89, in wrapper
    return func(*func_args, **func_kwargs)
  File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 70, in wrapper
    return f(*func_args, **func_kwargs)
  File "/opt/stack/tempest/tempest/scenario/test_network_basic_ops.py", line 920, in test_port_security_macspoofing_port
    self.check_remote_connectivity(ssh_client, dest=peer_address,
  File "/opt/stack/tempest/tempest/scenario/manager.py", line 1465, in check_remote_connectivity
    self.fail(msg)
  File "/usr/lib/python3.10/unittest/case.py", line 675, in fail
    raise self.failureException(msg)
AssertionError: Timed out waiting for 10.1.0.21 to become reachable from 172.24.5.84

[Lajos Katona (lajos-katona)]

In opensearch there are one example from this week:
https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_a=(columns:!(build_status,build_name),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22line%20920,%20in%20test_port_security_macspoofing_port%22'),sort:!())

[yatin (yatinkarel)]

It's seen quite frequently in tempest-slow-py3 and tempest-full-parallel(periodic so less execution) jobs. As per opensearch in last 54 hits:-

tempest-slow-py3 74.1%
tempest-full-parallel 12.0%
neutron-ovn-tempest-slow 7.4%
tempest-slow-2023-1 5.6%
tempest-all 0.9%

[yatin (yatinkarel)]

This looks to be a side effect of dhcpcd(configure all interfaces by default) available in cirros-0.6.0+ and the issue started to appear after jobs switched to use the newer cirros versions[1][2]

"test_port_security_macspoofing_port" is specifically impacted as it does manage the ip address manually, there was a slight change after the newer cirros version to it[3]

We could stop dhcpcd or reconfigure(denyinterfaces eth1 to /etc/dhcpcd.conf) so it do not interfare with the test config. And for this i found an issue in cirros and reported[4]

To fix we can update test as above or move affected jobs to use cirros 0.5.2 temporary.

[1] https://review.opendev.org/c/openstack/neutron/+/871272
[2] https://review.opendev.org/c/openstack/devstack/+/881437
[3] https://review.opendev.org/c/openstack/tempest/+/871271
[4] https://github.com/cirros-dev/cirros/issues/107

[yatin (yatinkarel)]

Proposed https://review.opendev.org/c/openstack/tempest/+/889713/

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 36.0.0 release.