Bug #2007166 “[tempest] CI job “neutron-ovn-tempest-ipv6-only-ov...” : Bugs : ovn package : Ubuntu

Rodolfo Alonso (rodolfo-alonso-hernandez) on 2023-02-13

Changed in neutron:
importance:	Undecided → Critical

yatin (yatinkarel) on 2023-02-14

Changed in neutron:
status:	New → Confirmed
tags:	added: gate-failure ovn tempest

yatin (yatinkarel) on 2023-02-14

Changed in neutron:
assignee:	nobody → yatin (yatinkarel)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-14: Related fix proposed to neutron (master)

#1

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/873684

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-15: Related fix merged to neutron (master)

#2

Reviewed: https://review.opendev.org/c/openstack/neutron/+/873684
Committed: https://opendev.org/openstack/neutron/commit/5dac5f8c6c4186921a7553116c38f5f03310b1d2
Submitter: "Zuul (22348)"
Branch: master

commit 5dac5f8c6c4186921a7553116c38f5f03310b1d2
Author: yatinkarel <email address hidden>
Date: Tue Feb 14 21:58:16 2023 +0530

[ovn-ipv6] Temporary exclude test_hotplug_nic

The test is failing quite often with ovn
version(v22.03.0) included in ubuntu jammy.

Until ubuntu jammy includes fixed version
of ovn let's exclude this test.

It's running fine on ubuntu focal and
ubuntu jammy jobs running with ovn from source.

Related-Bug: #2007166
Change-Id: I0377418bb5b4a68f9c08326bff0211f668a22b58

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-17: Related fix proposed to neutron (master)

#3

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/874253

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-20: Change abandoned on neutron (master)

#4

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/874253
Reason: Superseded by https://review.opendev.org/c/openstack/neutron/+/874112

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-02-20: Related fix merged to neutron (master)

#5

Reviewed: https://review.opendev.org/c/openstack/neutron/+/874112
Committed: https://opendev.org/openstack/neutron/commit/f991fc2fbef6b69b97d373e7acff8038b2da3028
Submitter: "Zuul (22348)"
Branch: master

commit f991fc2fbef6b69b97d373e7acff8038b2da3028
Author: yatinkarel <email address hidden>
Date: Thu Feb 16 21:19:24 2023 +0530

[ovn][ipv6] Skip test_update_delete_extra_route

    The issue is random and when it reproduces, any test
    involving localport(metadata access) ran just after
    test_update_delete_extra_route fails as metadata
    flows are missing.

The issue do not reproduce in ovn-22.06+(since [1]).

So until issue's fix is available in Ubuntu jammy,
we can skip this test.

[1] https://github.com/ovn-org/ovn/commit/eaf9832be2

    Related-Bug: #2007565
    Related-Bug: #2007166
    Change-Id: Ie2f5ff169bb3007694b9858465125775cfee40f4

Revision history for this message

yatin (yatinkarel) wrote on 2023-02-21:

#6

Reported Bug against OVN to track actual fix https://bugzilla.redhat.com/show_bug.cgi?id=2172036

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-06-01: Related fix proposed to neutron (master)

#7

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/885074

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-06-01: Related fix merged to neutron (master)

#8

Reviewed: https://review.opendev.org/c/openstack/neutron/+/885074
Committed: https://opendev.org/openstack/neutron/commit/2a8c7ff4f0a27d426db6d993b373a8222372abbd
Submitter: "Zuul (22348)"
Branch: master

commit 2a8c7ff4f0a27d426db6d993b373a8222372abbd
Author: yatinkarel <email address hidden>
Date: Thu Jun 1 21:25:37 2023 +0530

[ovn][ipv6] Skip test_update_router_admin_state

This is same to already skipped test in [1].

[1] https://review.opendev.org/c/openstack/neutron/+/874112

Related-Bug: #2007166
Change-Id: Iafc596e1ff13c7efde5c7da417db85bb562aa3f7

Revision history for this message

yatin (yatinkarel) wrote on 2023-06-22:

#9

Download full text (4.6 KiB)

Seen a few more failures:-
- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a63/886231/1/check/neutron-ovn-tempest-ipv6-only-ovs-release/a63687d/testr_results.html
- https://a3e174dd787d1569c081-614f3af0609b64225912d2a4919c81df.ssl.cf5.rackcdn.com/882581/7/check/neutron-ovn-tempest-ipv6-only-ovs-release/feb8b96/testr_results.html

$ grep admin_state_up screen-q-svc.txt |grep "'admin_state_up': True"|grep "Request body: {'router'"
Jun 16 16:28:00.617751 np0034320529 neutron-server[57489]: DEBUG neutron.api.v2.base [None req-2a66e63a-6f5e-4abd-9326-87edec5262f5 demo demo] Request body: {'router': {'admin_state_up': True, 'name': 'router1'}} {{(pid=57489) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 16:56:24.073625 np0034320529 neutron-server[57489]: DEBUG neutron.api.v2.base [None req-68e7d391-f799-42d6-a156-91566ee8a7d2 tempest-NetworksTestDHCPv6-1147368735 tempest-NetworksTestDHCPv6-1147368735-project-member] Request body: {'router': {'name': 'tempest-NetworksTestDHCPv6-router-863088563', 'external_gateway_info': {}, 'admin_state_up': True}} {{(pid=57489) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 16:56:29.565406 np0034320529 neutron-server[57489]: DEBUG neutron.api.v2.base [None req-69ecb9e9-adad-4bab-9d88-4d6a8ec93e16 tempest-NetworksTestDHCPv6-1147368735 tempest-NetworksTestDHCPv6-1147368735-project-member] Request body: {'router': {'name': 'tempest-NetworksTestDHCPv6-router-2081586294', 'external_gateway_info': {}, 'admin_state_up': True}} {{(pid=57489) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 16:58:48.759733 np0034320529 neutron-server[57489]: DEBUG neutron.api.v2.base [None req-e53ee47d-05b5-44d3-ae03-cd5c743e7c2a tempest-TestNetworkBasicOps-1899229111 tempest-TestNetworkBasicOps-1899229111-project-member] Request body: {'router': {'name': 'tempest-TestNetworkBasicOps-router-1236277632', 'admin_state_up': True, 'project_id': 'ffcabcbafb6248378ecf851ad0c4a820', 'external_gateway_info': {'network_id': '738da43f-f3f1-44eb-96b9-c00cba3b25bb'}}} {{(pid=57489) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 17:03:08.332661 np0034320529 neutron-server[57489]: DEBUG neutron.api.v2.base [None req-602e429e-3d8a-4a4e-a29c-6dbba2a9e2ba tempest-TestNetworkBasicOps-1899229111 tempest-TestNetworkBasicOps-1899229111-project-member] Request body: {'router': {'name': 'tempest-TestNetworkBasicOps-router-742545444', 'admin_state_up': True, 'project_id': 'ffcabcbafb6248378ecf851ad0c4a820', 'external_gateway_info': {'network_id': '738da43f-f3f1-44eb-96b9-c00cba3b25bb'}}} {{(pid=57489) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 17:05:31.764891 np0034320529 neutron-server[57490]: DEBUG neutron.api.v2.base [None req-43142176-2858-4515-8ed7-a4d2c97a934a tempest-TestSecurityGroupsBasicOps-146927725 tempest-TestSecurityGroupsBasicOps-146927725-project-member] Request body: {'router': {'name': 'tempest-TestSecurityGroupsBasicOps-router-162865443', 'admin_state_up': True, 'project_id': '9db58a07a2a14a5f82b6894431dae303', 'external_gateway_info': {'network_id...

Seen a few more failures:-
 -   https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a63/886231/1/check/neutron-ovn-tempest-ipv6-only-ovs-release/a63687d/testr_results.html
 -  https://a3e174dd787d1569c081-614f3af0609b64225912d2a4919c81df.ssl.cf5.rackcdn.com/882581/7/check/neutron-ovn-tempest-ipv6-only-ovs-release/feb8b96/testr_results.html

$ grep admin_state_up screen-q-svc.txt |grep "'admin_state_up': True"|grep "Request body: {'router'"
Jun 16 16:28:00.617751 np0034320529 neutron-server[57489]: DEBUG neutron.api.v2.base [None req-2a66e63a-6f5e-4abd-9326-87edec5262f5 demo demo] Request body: {'router': {'admin_state_up': True, 'name': 'router1'}} {{(pid=57489) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 16:56:24.073625 np0034320529 neutron-server[57489]: DEBUG neutron.api.v2.base [None req-68e7d391-f799-42d6-a156-91566ee8a7d2 tempest-NetworksTestDHCPv6-1147368735 tempest-NetworksTestDHCPv6-1147368735-project-member] Request body: {'router': {'name': 'tempest-NetworksTestDHCPv6-router-863088563', 'external_gateway_info': {}, 'admin_state_up': True}} {{(pid=57489) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 16:56:29.565406 np0034320529 neutron-server[57489]: DEBUG neutron.api.v2.base [None req-69ecb9e9-adad-4bab-9d88-4d6a8ec93e16 tempest-NetworksTestDHCPv6-1147368735 tempest-NetworksTestDHCPv6-1147368735-project-member] Request body: {'router': {'name': 'tempest-NetworksTestDHCPv6-router-2081586294', 'external_gateway_info': {}, 'admin_state_up': True}} {{(pid=57489) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 16:58:48.759733 np0034320529 neutron-server[57489]: DEBUG neutron.api.v2.base [None req-e53ee47d-05b5-44d3-ae03-cd5c743e7c2a tempest-TestNetworkBasicOps-1899229111 tempest-TestNetworkBasicOps-1899229111-project-member] Request body: {'router': {'name': 'tempest-TestNetworkBasicOps-router-1236277632', 'admin_state_up': True, 'project_id': 'ffcabcbafb6248378ecf851ad0c4a820', 'external_gateway_info': {'network_id': '738da43f-f3f1-44eb-96b9-c00cba3b25bb'}}} {{(pid=57489) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 17:03:08.332661 np0034320529 neutron-server[57489]: DEBUG neutron.api.v2.base [None req-602e429e-3d8a-4a4e-a29c-6dbba2a9e2ba tempest-TestNetworkBasicOps-1899229111 tempest-TestNetworkBasicOps-1899229111-project-member] Request body: {'router': {'name': 'tempest-TestNetworkBasicOps-router-742545444', 'admin_state_up': True, 'project_id': 'ffcabcbafb6248378ecf851ad0c4a820', 'external_gateway_info': {'network_id': '738da43f-f3f1-44eb-96b9-c00cba3b25bb'}}} {{(pid=57489) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 17:05:31.764891 np0034320529 neutron-server[57490]: DEBUG neutron.api.v2.base [None req-43142176-2858-4515-8ed7-a4d2c97a934a tempest-TestSecurityGroupsBasicOps-146927725 tempest-TestSecurityGroupsBasicOps-146927725-project-member] Request body: {'router': {'name': 'tempest-TestSecurityGroupsBasicOps-router-162865443', 'admin_state_up': True, 'project_id': '9db58a07a2a14a5f82b6894431dae303', 'external_gateway_info': {'network_id': '738da43f-f3f1-44eb-96b9-c00cba3b25bb'}}} {{(pid=57490) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 17:06:38.145410 np0034320529 neutron-server[57490]: DEBUG neutron.api.v2.base [None req-59dd9e6e-c2e6-46f5-b079-aacc92621cdd tempest-TestSecurityGroupsBasicOps-951623755 tempest-TestSecurityGroupsBasicOps-951623755-project-member] Request body: {'router': {'name': 'tempest-TestSecurityGroupsBasicOps-router-1380610968', 'admin_state_up': True, 'project_id': '0021dae7097e4c2c9ea837bef4dfb542', 'external_gateway_info': {'network_id': '738da43f-f3f1-44eb-96b9-c00cba3b25bb'}}} {{(pid=57490) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}
Jun 16 17:08:04.387898 np0034320529 neutron-server[57489]: DEBUG neutron.api.v2.base [None req-5fef2ce1-6dbb-484a-bfde-14cc65378e94 tempest-TestSecurityGroupsBasicOps-146927725 tempest-TestSecurityGroupsBasicOps-146927725-project-member] Request body: {'router': {'name': 'tempest-TestSecurityGroupsBasicOps-router-605452233', 'admin_state_up': True, 'project_id': '9db58a07a2a14a5f82b6894431dae303', 'external_gateway_info': {'network_id': '738da43f-f3f1-44eb-96b9-c00cba3b25bb'}}} {{(pid=57489) prepare_request_body /opt/stack/neutron/neutron/api/v2/base.py:731}}

Just before failing test routers are created/deleted for test test_dhcp_stateful_router[1], skipping it like other tests should help here

[1] https://opendev.org/openstack/tempest/src/branch/master/tempest/api/network/test_dhcp_ipv6.py#L405-L418

Revision history for this message

yatin (yatinkarel) wrote on 2023-08-01:

#10

Added Ubuntu(ovn) to impacted project. It would be good if OVN22.06 get's included in Ubuntu Jammy.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-12-21: Related fix proposed to neutron (stable/2023.1)

#11

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/904167

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-12-21:

#12

Related fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/neutron/+/904168

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-02: Related fix merged to neutron (stable/2023.1)

#13

Reviewed: https://review.opendev.org/c/openstack/neutron/+/904167
Committed: https://opendev.org/openstack/neutron/commit/a7cff2d2866a9319c20956ba3d3ac6a0a540b440
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit a7cff2d2866a9319c20956ba3d3ac6a0a540b440
Author: yatinkarel <email address hidden>
Date: Thu Jun 1 21:25:37 2023 +0530

[ovn][ipv6] Skip test_update_router_admin_state

This is same to already skipped test in [1].

[1] https://review.opendev.org/c/openstack/neutron/+/874112

    Related-Bug: #2007166
    Change-Id: Iafc596e1ff13c7efde5c7da417db85bb562aa3f7
    (cherry picked from commit 2a8c7ff4f0a27d426db6d993b373a8222372abbd)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-01-02:

#14

Reviewed: https://review.opendev.org/c/openstack/neutron/+/904168
Committed: https://opendev.org/openstack/neutron/commit/1d7d489b3a87170735ea59ccd8dfa9c87610d31b
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 1d7d489b3a87170735ea59ccd8dfa9c87610d31b
Author: yatinkarel <email address hidden>
Date: Wed Jun 28 13:19:44 2023 +0530

[ovn][ipv6] Add some more tests to skiplist

This is same to already skipped test in [1][2].

[1] https://review.opendev.org/c/openstack/neutron/+/874112
[2] https://review.opendev.org/c/openstack/neutron/+/885074

    Related-Bug: #2007166
    Change-Id: Ie93833538056edeb4da4c38e5139390f1c124724
    (cherry picked from commit add2a6eb8cd9507b666f3d2c289059c4e477e8e8)
    Conflicts:
            zuul.d/tempest-singlenode.yaml

Revision history for this message

Launchpad Janitor (janitor) wrote on 2024-04-26:

#15

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ovn (Ubuntu):
status:	New → Confirmed

Ubuntu
ovn package

[tempest] CI job "neutron-ovn-tempest-ipv6-only-ovs-release" unstable, most of the times because of "test_hotplug_nic"

Bug Description

Duplicates of this bug

Other bug subscribers

Remote bug watches

Affects		Status	Importance	Assigned to	Milestone
	neutron	Confirmed	Critical	yatin
	ovn (Ubuntu)	Confirmed	Undecided	Unassigned

Ubuntuovn package

[tempest] CI job "neutron-ovn-tempest-ipv6-only-ovs-release" unstable, most of the times because of "test_hotplug_nic"

Bug Description

Duplicates of this bug

Other bug subscribers

Remote bug watches

Ubuntu
ovn package