neutron

Metadata functional tests failing due to connection timeout

Bug #2079048 reported by Slawek Kaplonski on 2024-09-05

This bug report will be marked for expiration in 54 days if no further activity occurs. (find out why)

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Incomplete	High	Unassigned

Bug Description

Functional test "test_metadata_proxy_rate_limiting_ipv6" is failing from time to time due to the timeout while connecting to the metadata proxy (haproxy) service.

Examples of failure:

https://07237e8f4ab85c729718-eb12a13160f6c15076811ebfc6dacfe7.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-pyroute2-master/c8a8c8b/job-output.txt

https://9d8741f9b12c72356853-cbb980286bb4edf7e11c952127446228.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional/0ecf80c/job-output.txt

https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d2/926922/3/check/neutron-functional-with-uwsgi/6d257e7/job-output.txt

Stacktrace:

2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 2024-08-26 07:23:33.221508 | controller | Captured traceback:
07:23:33.221518 | controller | ~~~~~~~~~~~~~~~~~~~
07:23:33.221525 | controller | Traceback (most recent call last):
07:23:33.221572 | controller |
07:23:33.221583 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 109, in _query_metadata_proxy
07:23:33.221594 | controller | raw_headers = machine.execute(cmd)
07:23:33.221603 | controller |
07:23:33.221623 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/machine_fixtures.py", line 56, in execute
07:23:33.221636 | controller | return ns_ip_wrapper.netns.execute(*args, **kwargs)
07:23:33.221647 | controller |
07:23:33.221658 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 775, in execute
07:23:33.221672 | controller | return utils.execute(cmd, check_exit_code=check_exit_code,
07:23:33.221711 | controller |
07:23:33.221724 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py", line 155, in execute
07:23:33.221735 | controller | raise exceptions.ProcessExecutionError(msg,
07:23:33.221746 | controller |
07:23:33.222884 | controller | neutron_lib.exceptions.ProcessExecutionError: Exit code: 28; Cmd: ['ip', 'netns', 'exec', 'test-e2ea0587-ad1e-44fd-8d6d-b6e5efb3c857', 'curl', '--max-time', 60, '-D-', 'http://[fe80::a9fe:a9fe%port19caee]:80']; Stdin: ; Stdout: ; Stderr: % Total % Received % Xferd Average Speed Time Time Time Current
07:23:33.224396 | controller | Dload Upload Total Spent Left Speed
07:23:33.224505 | controller |
0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:04 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:05 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:06 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:07 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:08 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:09 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:10 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:11 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:12 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:13 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:14 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:15 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:16 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:17 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:18 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:19 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:20 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:21 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:22 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:23 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:24 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:25 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:26 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:27 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:28 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:30 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:31 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:32 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:33 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:34 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:35 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:36 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:37 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:38 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:39 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:40 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:41 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:42 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:43 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:44 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:45 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:46 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:47 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:48 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:49 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:50 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:51 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:52 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:53 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:54 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:55 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:56 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:57 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:58 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:00:59 --:--:-- 0
0 0 0 0 0 0 0 --:--:-- 0:01:00 --:--:-- 0
07:23:33.287025 | controller | curl: (28) Connection timed out after 60000 milliseconds
07:23:33.287075 | controller |
07:23:33.287087 | controller |
07:23:33.287096 | controller |
07:23:33.287105 | controller | During handling of the above exception, another exception occurred:
07:23:33.287112 | controller |
07:23:33.287122 | controller |
07:23:33.287131 | controller | Traceback (most recent call last):
07:23:33.287138 | controller |
07:23:33.287145 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 178, in func
07:23:33.287156 | controller | return f(self, *args, **kwargs)
07:23:33.287165 | controller |
07:23:33.287172 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 194, in test_metadata_proxy_rate_limiting_ipv6
07:23:33.287180 | controller | self._test_rate_limiting(2, machine, ipv6=True, interface=interface)
07:23:33.287190 | controller |
07:23:33.287199 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 169, in _test_rate_limiting
07:23:33.287206 | controller | firstline = self._query_metadata_proxy(machine, ipv6=ipv6,
07:23:33.287213 | controller |
07:23:33.287223 | controller | File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/functional/agent/l3/test_metadata_proxy.py", line 116, in _query_metadata_proxy
07:23:33.287231 | controller | self.fail('metadata proxy unreachable '
07:23:33.287239 | controller |
07:23:33.287245 | controller | File "/usr/lib/python3.10/unittest/case.py", line 675, in fail
07:23:33.287255 | controller | raise self.failureException(msg)
07:23:33.287264 | controller |
07:23:33.287271 | controller | AssertionError: metadata proxy unreachable on http://[fe80::a9fe:a9fe%port19caee]:80 before timeout

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-09-05: Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/928136

Revision history for this message

yatin (yatinkarel) wrote on 2024-09-09:

Another hit:- https://07237e8f4ab85c729718-eb12a13160f6c15076811ebfc6dacfe7.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/neutron-functional-with-pyroute2-master/c8a8c8b/testr_results.html

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-09-10: Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/928136
Committed: https://opendev.org/openstack/neutron/commit/642b2910326db233253ffbd876006b8193c8e167
Submitter: "Zuul (22348)"
Branch: master

commit 642b2910326db233253ffbd876006b8193c8e167
Author: Slawek Kaplonski <email address hidden>
Date: Tue Sep 10 10:39:51 2024 +0200

Enable iptables debugging in the L3 agent functional tests

This patch enables debug_iptables_rules config option in the L3 agent
related functional tests.

Related-Bug: #2079048
Change-Id: I6cb17b149626fb88c82937e046632dc8ca8d7ced

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-09-17: Related fix proposed to neutron-fwaas (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-fwaas/+/929595

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-09-19: Related fix merged to neutron-fwaas (master)

Reviewed: https://review.opendev.org/c/openstack/neutron-fwaas/+/929658
Committed: https://opendev.org/openstack/neutron-fwaas/commit/caca5ae4a0adbf5a2f2eeabbd746dac9d3ac37e6
Submitter: "Zuul (22348)"
Branch: master

commit caca5ae4a0adbf5a2f2eeabbd746dac9d3ac37e6
Author: Brian Haley <email address hidden>
Date: Tue Sep 17 10:58:57 2024 -0400

Account for iptables-save output spacing differences

There are places where the iptables-save output is not
exactly as the input, for example:

    1) extra space after '-j NFLOG --nflog-prefix'
    2) '#/sec' instead of '#/s' for limit-burst
    3) '-j REJECT --reject-with icmp-port-unreachable' instead
       of '-REJECT'

Account for that in the code so when iptables debug is
enabled the functional tests pass.

Related-bug: #2079048
Closes-bug: #2080933

Change-Id: I98fe93019b7d1b84d0622b4430e56b37b7cc0250

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-09-19: Change abandoned on neutron-fwaas (master)

Change abandoned by "Brian Haley <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron-fwaas/+/929595
Reason: https://review.opendev.org/c/openstack/neutron-fwaas/+/929658 was merged instead

Revision history for this message

yatin (yatinkarel) wrote on 2024-09-23:

Recent failure since logs added https://402540ecdd63d1ecb8cc-46682989e655431f9d8a4a56f6d0c4c2.ssl.cf1.rackcdn.com/929793/3/gate/neutron-functional-with-uwsgi/5a7d403/testr_results.html

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-09-24: Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/930336

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2024-09-24:

After investigation of the logs from failed test with iptables rules logged, I am sure that this issue is not caused by missing or wrong iptables rules.

I was however able to reproduce this issue locally couple of times (but very rare and it happens randomly) and it seems for me that the issue happens when link local IPv6 address (fe80::c8fe:deff:fead:beef/64) is missing from the qr- interface in the router namespace. To confirm that this is the same issue in u/s ci, there is new patch proposed https://review.opendev.org/c/openstack/neutron/+/930336 to add some extra logs when this test will fail.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-09-27: Related fix merged to neutron (master)

#10

Reviewed: https://review.opendev.org/c/openstack/neutron/+/930336
Committed: https://opendev.org/openstack/neutron/commit/770ce6150a80da6df518d966cc9598978cf57278
Submitter: "Zuul (22348)"
Branch: master

commit 770ce6150a80da6df518d966cc9598978cf57278
Author: Slawek Kaplonski <email address hidden>
Date: Tue Sep 24 15:57:44 2024 +0200

[Functional tests] Add logging router interfaces in metadata IPv6 tests

This is added to help understand the root cause of the issue with
communication with metadata server like e.g. described in the related bug.

Related-bug: #2079048
Change-Id: I5f6cfc4f8f25a82ca703c0d2b36c2de92fc1f20d

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-11-29: Related fix proposed to neutron-fwaas (stable/2024.2)

#11

Related fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/neutron-fwaas/+/936739

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2024-12-02: Related fix merged to neutron-fwaas (stable/2024.2)

#12

Reviewed: https://review.opendev.org/c/openstack/neutron-fwaas/+/936739
Committed: https://opendev.org/openstack/neutron-fwaas/commit/36db6ddb6a2369377bcf82b37067319304d075fd
Submitter: "Zuul (22348)"
Branch: stable/2024.2

commit 36db6ddb6a2369377bcf82b37067319304d075fd
Author: Brian Haley <email address hidden>
Date: Tue Sep 17 10:58:57 2024 -0400

Account for iptables-save output spacing differences

There are places where the iptables-save output is not
exactly as the input, for example:

    1) extra space after '-j NFLOG --nflog-prefix'
    2) '#/sec' instead of '#/s' for limit-burst
    3) '-j REJECT --reject-with icmp-port-unreachable' instead
       of '-REJECT'

Account for that in the code so when iptables debug is
enabled the functional tests pass.

Related-bug: #2079048
Closes-bug: #2080933

Change-Id: I98fe93019b7d1b84d0622b4430e56b37b7cc0250
(cherry picked from commit caca5ae4a0adbf5a2f2eeabbd746dac9d3ac37e6)

Revision history for this message

yatin (yatinkarel) wrote on 2025-01-27:

#13

latest it https://8a511f4596c1d3e1e6a3-ae6729a96e1d6a6e5a3d46e4b2a2210d.ssl.cf1.rackcdn.com/938911/1/gate/neutron-functional-with-uwsgi/f9fc2e5/testr_results.html

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2025-01-27: Related fix proposed to neutron (stable/2024.2)

#14

Related fix proposed to branch: stable/2024.2
Review: https://review.opendev.org/c/openstack/neutron/+/940250

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2025-01-27: Related fix proposed to neutron (stable/2024.1)

#15

Related fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/neutron/+/940251

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2025-01-27: Related fix proposed to neutron (stable/2023.2)

#16

Related fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/neutron/+/940252

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2025-01-27: Related fix merged to neutron (stable/2024.1)

#17

Reviewed: https://review.opendev.org/c/openstack/neutron/+/940251
Committed: https://opendev.org/openstack/neutron/commit/4df4313b7fe367f08a94505a39d693bad9036a35
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit 4df4313b7fe367f08a94505a39d693bad9036a35
Author: Slawek Kaplonski <email address hidden>
Date: Tue Sep 24 15:57:44 2024 +0200

[Functional tests] Add logging router interfaces in metadata IPv6 tests

This is added to help understand the root cause of the issue with
communication with metadata server like e.g. described in the related bug.

    Related-bug: #2079048
    Change-Id: I5f6cfc4f8f25a82ca703c0d2b36c2de92fc1f20d
    (cherry picked from commit 770ce6150a80da6df518d966cc9598978cf57278)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2025-01-28: Related fix merged to neutron (stable/2024.2)

#18

Reviewed: https://review.opendev.org/c/openstack/neutron/+/940250
Committed: https://opendev.org/openstack/neutron/commit/318f2df7f51b85aafcb4e78fb66ebb12bde4f95a
Submitter: "Zuul (22348)"
Branch: stable/2024.2

commit 318f2df7f51b85aafcb4e78fb66ebb12bde4f95a
Author: Slawek Kaplonski <email address hidden>
Date: Tue Sep 24 15:57:44 2024 +0200

[Functional tests] Add logging router interfaces in metadata IPv6 tests

This is added to help understand the root cause of the issue with
communication with metadata server like e.g. described in the related bug.

    Related-bug: #2079048
    Change-Id: I5f6cfc4f8f25a82ca703c0d2b36c2de92fc1f20d
    (cherry picked from commit 770ce6150a80da6df518d966cc9598978cf57278)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2025-01-29: Related fix merged to neutron (stable/2023.2)

#19

Reviewed: https://review.opendev.org/c/openstack/neutron/+/940252
Committed: https://opendev.org/openstack/neutron/commit/527ca52880f4d9c400e88fc05cd24dfd819e3bcf
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit 527ca52880f4d9c400e88fc05cd24dfd819e3bcf
Author: Slawek Kaplonski <email address hidden>
Date: Tue Sep 24 15:57:44 2024 +0200

[Functional tests] Add logging router interfaces in metadata IPv6 tests

This is added to help understand the root cause of the issue with
communication with metadata server like e.g. described in the related bug.

    Related-bug: #2079048
    Change-Id: I5f6cfc4f8f25a82ca703c0d2b36c2de92fc1f20d
    (cherry picked from commit 770ce6150a80da6df518d966cc9598978cf57278)

Revision history for this message

Slawek Kaplonski (slaweq) wrote on 2025-05-16:

#20

I'm closing it now as we didn't saw new occurences of similar issues for quite some time.

Changed in neutron:
status:	Confirmed → Incomplete

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.