MacvtapAgentTestCase test_get_all_devices test is failing due to MAC addresses mismatch

[Slawek Kaplonski]

Failure examples:

    https://8ea5d1c610c25577a750-2d27e805ff456f7d0a585c8eee091a7d.ssl.cf5.rackcdn.com/841838/69/check/neutron-functional-with-uwsgi/9ab3804/testr_results.html

    https://e5b18301b2a8acb0dbed-61111cd830f97ca8dcba466112f0eafd.ssl.cf5.rackcdn.com/863881/9/check/neutron-functional-with-uwsgi/60e2715/testr_results.html

Stacktrace:

ft1.1: neutron.tests.functional.plugins.ml2.drivers.macvtap.agent.test_macvtap_neutron_agent.MacvtapAgentTestCase.test_get_all_devicestesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/plugins/ml2/drivers/macvtap/agent/test_macvtap_neutron_agent.py", line 36, in test_get_all_devices
    self.assertEqual(set([macvtap.link.address]),
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 393, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional-gate/lib/python3.10/site-packages/testtools/testcase.py", line 480, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: {'82:78:e1:db:c9:07'} != {'d6:1e:f1:39:2c:71'}

[Lajos Katona (lajos-katona)]

Seems like it started ~around 11. 20.:
https://opensearch.logs.openstack.org/_dashboards/app/discover/?security_tenant=global#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-5d,to:now))&_a=(columns:!(build_status,build_name),filters:!(),index:'94869730-aea8-11ec-9e6a-83741af3fdcd',interval:auto,query:(language:kuery,query:'message:%22test_get_all_devices%22'),sort:!())

(Sorry for bad filter, but this worked at least...)
It seems to be close to the ml-1 release dates so perhaps that can be a good first check.

[Lajos Katona (lajos-katona)]

I can't reproduce it on my local env (jammy)....

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

Thanks Lajos for the log filter. The first occurrence of this issue is on "Nov 20, 2022 @ 13:11:42.120", according to this list. That matches with [1], that was merged 3 days ago (Nov 19). Neutron functional inherits from "devstack-minimal" and now this job is being executed in Ubuntu 22.04. From a failing job [2]: https://paste.opendev.org/show/b9mlq479UpMxgjMjSdgy/

I'll try to reproduce this in this OS version.

Regards.

[1]https://review.opendev.org/c/openstack/devstack/+/860795/9/.zuul.yaml#488
[2]https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_840/860765/7/check/neutron-functional-with-uwsgi/8402bf7/job-output.txt

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/865414

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/865414
Committed: https://opendev.org/openstack/neutron/commit/ff66ca92755970c88fbb0f347400786089c11c4b
Submitter: "Zuul (22348)"
Branch: master

commit ff66ca92755970c88fbb0f347400786089c11c4b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Nov 23 11:19:31 2022 +0000

    Execute functional test "test_get_all_devices" alone

    Since [1], the functional tests are being executed in Ubuntu 22.04.
    The test "test_get_all_devices" is failing frequently, maybe because
    other tests are interfering and modifying the root namespace
    interface list.

    Because this test cannot be executed inside a temporary namespace,
    this patch moves the execution of this test to the section of
    "concurrency=1". That will ensure no other test interferes the
    result.

    Closes-Bug: #1997341

    Change-Id: If0631bf0cdfd5f0989c60d40e31f89cee317e497

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.0.0.0rc1 release candidate.