[functional/fullstack][master] move fips periodic job to CentOS 9-Stream

[yatin]

Currently these periodic jobs are running on CentOS 8-Stream(with python 3.6) and failing[1][2]. These are failing as master no longer supports py3.6. To unblock switching these jobs to run functional/fullstack tests with python3.8[3] and disabling dbcounter installation.

Ideally these jobs can be switched to CentOS 9-Stream(have python3.9 as default). But found in testing[4] it has couple of issues:-
1. DNS issues are hit randomly during setup until unbound service is setupped after reboot. Can fix by waiting for unbound to be ready or workaround by not using unbound.
2. 4 functional tests are failing:-
- test_delete_multiple_entries --> conntrack delete not deleting enteries (WARNING neutron.privileged.agent.linux.netlink_lib [-] Netlink query failed looks related)
- test_delete_icmp_entry --> conntrack delete not deleting enteries (WARNING neutron.privileged.agent.linux.netlink_lib [-] Netlink query failed looks related)
- test_rule_application_converges(IptablesFirewallDriver,with ipset)
   --> self.assertEqual([], self.firewall.iptables._apply()) fails as _apply not returning empty list as expected --> might be due to iptables behavior in C9-Stream
- test_rule_application_converges(IptablesFirewallDriver,without ipset)
   --> self.assertEqual([], self.firewall.iptables._apply()) fails as _apply not returning empty list as expected --> might be due to iptables behavior in C9-Stream

3. 2 fullstack tests are failing:-
- neutron.tests.fullstack.test_l3_agent.TestLegacyL3Agent.test_north_south_traffic
- neutron.tests.fullstack.test_local_ip.LocalIPTestCase.test_vm_is_accessible_by_local_ip(static_nat)
Both failing at ping
neutron.tests.common.machine_fixtures.FakeMachineException: No ICMP reply obtained from IP address 2001:db8:1234::
neutron.tests.common.machine_fixtures.FakeMachineException: No ICMP reply obtained from IP address 10.0.0.10

[1] https://zuul.openstack.org/builds?job_name=neutron-functional-with-uwsgi-fips&project=openstack%2Fneutron&branch=master&pipeline=periodic&skip=0
[2] https://zuul.openstack.org/builds?job_name=neutron-functional-with-uwsgi-fips&project=openstack%2Fneutron&branch=master&pipeline=periodic&skip=0
[3] https://review.opendev.org/c/openstack/neutron/+/843252
[4] https://review.opendev.org/c/openstack/neutron/+/843245

[yatin (yatinkarel)]

Switch fips functional/fullstack job to py3.8 https://review.opendev.org/c/openstack/neutron/+/843252

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/843252
Committed: https://opendev.org/openstack/neutron/commit/7f5fb86232a92008e3f9c3be8a00287a1cd09b20
Submitter: "Zuul (22348)"
Branch: master

commit 7f5fb86232a92008e3f9c3be8a00287a1cd09b20
Author: yatinkarel <email address hidden>
Date: Wed May 25 14:42:00 2022 +0530

    Switch Fips fullstack/functional tests to py3.8

    Master has dropped py3.6 support so upper-constraints
    requirements are not installable with python3.6
    and due to this fips jobs which are running on py3.6
    are failing. Let's run the functional/fullstack tests
    with python3.8 on CentOS 8-Stream itself.

    Also disable dbcounter installation as it's not
    installing on python3.6.
    The hacks can be cleaned up when these jobs are
    switched to CentOS 9-Stream, currently have some
    issues as mentioned in the Related Bug.

    Depends-On: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/843989
    Related-Bug: #1976323
    Change-Id: I757dbf929cb1dc27027614bbb2147a61606f85a9

[Slawek Kaplonski (slaweq)]

Functional tests are failing on centos9 stream even without FIPS enabled so it's not FIPS related issue. I will continue investigation why those tests are failing there.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/844589

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/844589
Committed: https://opendev.org/openstack/neutron/commit/6c2d2744d5e841f431ae637624dd5954429f9607
Submitter: "Zuul (22348)"
Branch: master

commit 6c2d2744d5e841f431ae637624dd5954429f9607
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jul 27 20:48:45 2022 +0200

    Make iptables rules diff to be case insensitive

    MAC addresses in the iptables rules are always added by iptables_manager
    with uppercase. It was done like that in [1].
    For some reason, iptables on Centos 9 Stream (1.8.7) returns MAC
    addresses all in lowercase and difflib library treats such new and old
    rules as different ones so iptables manager always tries to remove and
    add antispoofing rules which have MAC addresses set.

    [1] https://review.opendev.org/c/openstack/neutron/+/184355

    Related-Bug: #1976323
    Change-Id: I63e78fdd030f063a1b943d481a9cbd98850552d7

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/843245
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[yatin (yatinkarel)]

Functional job is green as seen in test patch https://review.opendev.org/c/openstack/neutron/+/843245

<< 3. 2 fullstack tests are failing:-

I checked the fullstack failures and found [1] which made the issue[2] appear in CentOS 9 Stream.
Ubuntu jammy didn't have the issue as it has the fixes(ping succeed but add warning DIFFERENT ADDRESS) available[3][4][5], i verified by using a build from fedora[6]

Version in CentOS 9-stream:- iputils-20210202-8.el9.x86_64
Version in Ubuntu jammy:- 3:20211215-1

Other observations which i am not sure if expected or not
neutron.tests.fullstack.test_local_ip.LocalIPTestCase.test_vm_is_accessible_by_local_ip(static_nat):- ping from vm1 to vm0, response comes from local_ip assigned to vm0
- after arp cache expiry of 30 seconds ping response works fine

neutron.tests.fullstack.test_l3_agent.TestLegacyL3Agent.test_north_south_traffic
ping from vm1 or qrouter namespace to external gateway(2001:db8:1234::), response comes from router interface

[1] https://github.com/iputils/iputils/commit/5e052ada96c3fc457efe66b2fed5c14ed8de6ae3
[2] https://github.com/iputils/iputils/issues/371
[3] https://github.com/iputils/iputils/commit/15a5e5c7aace5a7a782ff802988e04ed4c1148a5
[4] https://github.com/iputils/iputils/commit/5f6bec5ab57cc8beaa78f5756a0ffbdf01f28d36
[5] https://github.com/iputils/iputils/commit/dd0bcac98d10ea340dfe8aff43f26fe1a7f77931
[6] https://koji.fedoraproject.org/koji/buildinfo?buildID=1868148

[Oleg Bondarev (obondarev)]

>>neutron.tests.fullstack.test_local_ip.LocalIPTestCase.test_vm_is_accessible_by_local_ip(static_nat):- ping from vm1 to vm0, response comes from local_ip assigned to vm0
>> - after arp cache expiry of 30 seconds ping response works fine

I believe this is a side effect/bug in Local IP feature, so 30 seconds workaround seems good short term solution, but ideally should be investigated and fixed in Local IP itself

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 23.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/894857

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/895023

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/894857
Committed: https://opendev.org/openstack/neutron/commit/b7a59a51320ace612276d35c0358e7d5ee90de8e
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit b7a59a51320ace612276d35c0358e7d5ee90de8e
Author: yatinkarel <email address hidden>
Date: Wed May 25 13:18:44 2022 +0530

    Switch fullstack/functional fips jobs to 9-stream

    Master no longer support py3.6, so let's switch
    these jobs to CentOS 9-stream which includes py3.9.

    Also dbcounter[1] is not installable on CentOS 8-stream
    and hence these jobs are currently broken.

    Other fips jobs already switched with[2].

    [1] https://review.opendev.org/c/openstack/devstack/+/839820
    [2] https://review.opendev.org/c/openstack/neutron/+/833173

    [Stable Only] The switch is to match defined runtimes per release
    https://governance.openstack.org/tc/reference/runtimes/

    Closes-Bug: #1976323
    Change-Id: Ie4e807b1490d59390316ec20b499b7676acfe410
    (cherry picked from commit 42ae9448701b7925b736f5706140d414b53d9012)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895023
Committed: https://opendev.org/openstack/neutron/commit/a48f34022da49e34f1b5639271407840cd609788
Submitter: "Zuul (22348)"
Branch: stable/zed

commit a48f34022da49e34f1b5639271407840cd609788
Author: yatinkarel <email address hidden>
Date: Wed May 25 13:18:44 2022 +0530

    Switch fullstack/functional fips jobs to 9-stream

    Master no longer support py3.6, so let's switch
    these jobs to CentOS 9-stream which includes py3.9.

    Also dbcounter[1] is not installable on CentOS 8-stream
    and hence these jobs are currently broken.

    Other fips jobs already switched with[2].

    [1] https://review.opendev.org/c/openstack/devstack/+/839820
    [2] https://review.opendev.org/c/openstack/neutron/+/833173

    [Stable Only] The switch is to match defined runtimes per release
    https://governance.openstack.org/tc/reference/runtimes/

    Closes-Bug: #1976323
    Change-Id: Ie4e807b1490d59390316ec20b499b7676acfe410
    (cherry picked from commit 42ae9448701b7925b736f5706140d414b53d9012)
    (cherry picked from commit b7a59a51320ace612276d35c0358e7d5ee90de8e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 22.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.2.0 release.