Queens, Rocky, Stein, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing. Timeout
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
tripleo |
Fix Released
|
Critical
|
Amol Kahat |
Bug Description
Description
===========
- Network tests are failing on the queens jobs.
Tests:
- tempest.
Traceback:
Traceback (most recent call last):
File "/usr/lib/
return f(*func_args, **func_kwargs)
File "/usr/lib/
self.
File "/usr/lib/
self.servers, mtu=mtu)
File "/usr/lib/
mtu=mtu)
File "/usr/lib/
msg=msg)
File "/usr/lib/
raise self.failureExc
AssertionError: False is not true : Timed out waiting for 192.168.24.105 to become reachable
Pipeline
========
- openstack-
Affected Jobs
=============
- periodic-
- periodic-
- periodic-
- periodic-
Logs
====
- https:/
- https:/
- https:/
- https:/
summary: |
- Network tests are failing on queens jobs + Queens, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps + failing. Timeout |
Changed in tripleo: | |
milestone: | none → victoria-1 |
summary: |
- Queens, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps - failing. Timeout + Queens, Rocky, Stein, + tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing. + Timeout |
This is really hard to identify, but I believe it's related to network, more specifically the openvswitch.
I checked the logs for neutron. The requests were executed properly, no failures, but on openvswitch log, I see the following errors:
2020-04-30 10:06:37.281 80095 DEBUG neutron. plugins. ml2.drivers. openvswitch. agent.ovs_ neutron_ agent [req-0274982a- 50f5-4940- 88e9-9eca4ee530 99 - - - - -] Configuring tunnel endpoints to other OVS agents tunnel_sync /usr/lib/ python2. 7/site- packages/ neutron/ plugins/ ml2/drivers/ openvswitch/ agent/ovs_ neutron_ agent.py: 1930 agent.linux. async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Bridge name --format=json]: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory agent.linux. async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Bridge name --format=json]: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5) agent.linux. async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Bridge name --format=json]: PMD: net_mlx4: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory agent.linux. async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Bridge name --format=json]: PMD: net_mlx4: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx4) agent.linux. async_process [-] Output received from [ovsdb-client monitor tcp:127.0.0.1:6640 Bridge name --format=json]: {"data" :[["40be3a55- a226-4324- 904c-7b085dcd66 70","initial" ,"br-ex" ],["6cc79323- 4dd0-4ad0- 8b57-0ea869d51a 6c","initial" ,"br-int" ],["c24511d9- 33c5-445f- ad84-206519bf03 f1","initial" ,"br-tun" ]],"headings" :["row" ,"action" ,"name" ]} _read_stdout /usr/lib/ python2. 7/site- packages/ neutron/ agent/linux/ async_process. py:260
2020-04-30 10:06:37.283 80095 ERROR neutron.
2020-04-30 10:06:37.283 80095 ERROR neutron.
2020-04-30 10:06:37.283 80095 ERROR neutron.
2020-04-30 10:06:37.284 80095 ERROR neutron.
2020-04-30 10:06:37.285 80095 DEBUG neutron.
If it is relevant for this bug or not, only the ovs guys can tell for sure.
This error happens before the test be executed, as you can notice that the first request from the test to setup network is the req-2e78b6d1- f293-47cf- 9861-9316b92d5a 04, but perhaps is affecting how the iptables/routes are handled.
I also found some interesting logs here:
'neutron. api.rpc. agentnotifiers. dhcp_rpc_ agent_api. DhcpAgentNotify API._native_ event_send_ dhcp_notificati on--92233720368 54012965' , 'neutron. services. segments. plugin. NovaSegmentNoti fier._notify_ port_created_ or_deleted- -92233720368542 48080', 'neutron. plugins. ml2.plugin. Ml2Plugin. notify_ sg_on_port_ change- 66733164' ] for port, after_create _notify_loop /usr/lib/ python2. 7/site- packages/ neutron_ lib/callbacks/ manager. py:167 lib.callbacks. manager [req-a4c2abfc- c635-4eb6- a547-eb88bba5c9 52 - - - - -] Notify callbacks ['neutron. services. qos.qos_ plugin. QoSPlugin. _validate_ create_ port_callback- -92233720368546 76145'] for port, precommit_create _notify_loop /usr/lib/pyth...
2020-04-30 10:10:31.445 30 DEBUG neutron_