fedora28 standalone failing on tempest

Bug #1813224 reported by Alex Schultz on 2019-01-24
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Undecided
Brian Haley
tripleo
Critical
Arx Cruz

Bug Description

The fedora28 tempest jobs are failing in check (it's voting but not in the gate).

tempest.scenario.test_network_basic_ops.TestNetworkBasicOps
tempest.scenario.test_server_basic_ops.TestServerBasicOps
tempest.scenario.test_minimum_basic.TestMinimumBasicScenario

http://logs.openstack.org/31/626631/11/check/tripleo-ci-fedora-28-standalone/cf314a4/logs/undercloud/home/zuul/tempest/tempest.html.gz

http://logs.openstack.org/53/623353/10/check/tripleo-ci-fedora-28-standalone/0841969/logs/tempest.html

http://logs.openstack.org/56/593056/43/check/tripleo-ci-fedora-28-standalone/106db25/logs/tempest.html

http://logs.openstack.org/97/631297/3/check/tripleo-ci-fedora-28-standalone/7fe7dc1/logs/tempest.html

sova is reporting this job to be <80% passing.

Tags: ci Edit Tag help
Alex Schultz (alex-schultz) wrote :

review to revert the voting status https://review.openstack.org/#/c/633087/

wes hayutin (weshayutin) wrote :

Arx will be looking into this..

Changed in tripleo:
assignee: nobody → Arx Cruz (arxcruz)
Emilien Macchi (emilienm) wrote :

removing the alert, as the job isn't voting anymore.

tags: removed: alert
Arx Cruz (arxcruz) wrote :
Download full text (3.5 KiB)

For this particular case, I found this error:

2019-01-24 19:01:33.969 9 ERROR vif_plug_ovs.ovsdb.impl_vsctl [req-9ba4d327-4bad-4a45-99c1-9dcf22b37193 f14d275e92cb4384ba0502ca1738c8cc 3879640c10fe4962a8ca60cd0be8dbd9 - default default] Unable to execute ['ovs-vsctl', '--timeout=120', '--oneline', '--format=json', '--', '--may-exist', 'add-br', u'br-int', '--', 'set', 'Bridge', u'br-int', u'datapath_type=system']. Exception: Unexpected error while running command.
Command: ovs-vsctl --timeout=120 --oneline --format=json -- --may-exist add-br br-int -- set Bridge br-int datapath_type=system
Exit code: 1
Stdout: u''
Stderr: u'ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n': ProcessExecutionError: Unexpected error while running command.
2019-01-24 19:01:33.995 9 ERROR vif_plug_ovs.ovsdb.impl_vsctl [req-9ba4d327-4bad-4a45-99c1-9dcf22b37193 f14d275e92cb4384ba0502ca1738c8cc 3879640c10fe4962a8ca60cd0be8dbd9 - default default] Unable to execute ['ovs-vsctl', '--timeout=120', '--oneline', '--format=json', '--', '--may-exist', 'add-port', u'br-int', u'qvo94387b07-f4', '--', 'set', 'Interface', u'qvo94387b07-f4', u'external_ids:iface-id=94387b07-f47c-4529-a398-4e09a6f7c7ce', 'external_ids:iface-status=active', u'external_ids:vm-uuid=67d0ffd8-13ca-4271-9eeb-fd16ae96bbad', u'external_ids:attached-mac=fa:16:3e:9e:cb:9f']. Exception: Unexpected error while running command.
Command: ovs-vsctl --timeout=120 --oneline --format=json -- --may-exist add-port br-int qvo94387b07-f4 -- set Interface qvo94387b07-f4 external_ids:iface-id=94387b07-f47c-4529-a398-4e09a6f7c7ce external_ids:iface-status=active external_ids:vm-uuid=67d0ffd8-13ca-4271-9eeb-fd16ae96bbad external_ids:attached-mac=fa:16:3e:9e:cb:9f
Exit code: 1
Stdout: u''
Stderr: u'ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)\n': ProcessExecutionError: Unexpected error while running command.

http://logs.openstack.org/31/626631/11/check/tripleo-ci-fedora-28-standalone/cf314a4/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz#_2019-01-24_19_01_33_969

Checking other jobs, I also found this:

2019-01-27 20:05:04.762 9 ERROR vif_plug_ovs.ovsdb.impl_vsctl [req-c0565eb3-be73-4d2a-b59e-a7cc37f759d5 c9745094b10545779ae32bf53dd8cca9 7afb8674582846bc82865e1a71d6e67c - default default] Unable to execute ['ovs-vsctl', '--timeout=120', '--oneline', '--format=json', '--', '--may-exist', 'add-port', u'br-int', u'qvo1a3b80e5-55', '--', 'set', 'Interface', u'qvo1a3b80e5-55', u'external_ids:iface-id=1a3b80e5-5521-4134-bad5-3841ec13eead', 'external_ids:iface-status=active', u'external_ids:vm-uuid=4d420cb6-dc79-48d6-9df3-2d7041f8ca05', u'external_ids:attached-mac=fa:16:3e:78:78:8f']. Exception: Unexpected error while running command.
Command: ovs-vsctl --timeout=120 --oneline --format=json -- --may-exist add-port br-int qvo1a3b80e5-55 -- set Interface qvo1a3b80e5-55 external_ids:iface-id=1a3b80e5-5521-4134-bad5-3841ec13eead external_ids:iface-status=active external_ids:vm-uuid=4d420cb6-dc79-48d6-9df3-2d7041f8ca05 external_ids:attached-mac=fa:16:3e:78:78:8f
Exit code: -14
Stdout: u''
Stderr: u'2019-01-27T...

Read more...

Arx Cruz (arxcruz) wrote :

Openvswitch version installed: openvswitch-2.10.1-1.fc28.x86_64

Miguel Angel Ajo (mangelajo) wrote :

There are lots of warnings showing up in ovs-vswitchd log

2019-01-24T18:40:59.070Z|00058|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: 2 flow_mods 10 s ago (2 adds)
2019-01-24T18:40:59.184Z|00059|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: 10 flow_mods 10 s ago (10 adds)
2019-01-24T18:46:19.496Z|00060|bridge|INFO|bridge br-int: added interface tapd0b4fd96-39 on port 3
2019-01-24T18:46:19.609Z|00061|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on tapd0b4fd96-39 device failed: No such device
2019-01-24T18:46:25.160Z|00062|bridge|INFO|bridge br-int: added interface qr-32fda31e-3a on port 4
2019-01-24T18:46:25.272Z|00063|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qr-32fda31e-3a device failed: No such device
2019-01-24T18:46:25.645Z|00064|bridge|INFO|bridge br-int: added interface qg-193495cd-3d on port 5
2019-01-24T18:46:25.701Z|00065|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qg-193495cd-3d device failed: No such device

Changed in neutron:
assignee: nobody → Brian Haley (brian-haley)
milestone: none → stein-3
Brian Haley (brian-haley) wrote :

So I see errors like this in one of the logs, http://logs.openstack.org/97/631297/3/check/tripleo-ci-fedora-28-standalone/7fe7dc1/logs/undercloud/var/log/containers/neutron/openvswitch-agent.log.txt.gz

2019-01-24 17:01:14.064 56271 ERROR neutron.agent.common.ovsdb_monitor [req-6c83cba5-d914-4ad1-a4de-6368051abd6b - - - - -] Interface monitor is not active
...
2019-01-24 17:01:14.534 56271 ERROR neutron.agent.common.async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: PMD: net_mlx5: cannot load glue library: libibverbs.so.1: cannot open shared object file: No such file or directory
2019-01-24 17:01:14.536 56271 ERROR neutron.agent.common.async_process [-] Error received from [ovsdb-client monitor tcp:127.0.0.1:6640 Interface name,ofport,external_ids --format=json]: PMD: net_mlx5: cannot initialize PMD due to missing run-time dependency on rdma-core libraries (libibverbs, libmlx5)

And things like this in the l3-agent, http://logs.openstack.org/97/631297/3/check/tripleo-ci-fedora-28-standalone/7fe7dc1/logs/undercloud/var/log/containers/neutron/l3-agent.log.txt.gz

19-01-24 17:01:11.623 55617 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1337 command(idx=1): DbSetCommand(table=Port, col_values=(('tag', 4095),), record=qg-49c21fc0-4e) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-01-24 17:01:11.623 55617 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1337 command(idx=2): DbSetCommand(table=Interface, col_values=(('type', 'internal'), ('external_ids', {'iface-id': u'49c21fc0-4ee5-4c45-a0f6-971c74601d17', 'iface-status': 'active', 'attached-mac': u'fa:16:3e:7c:c1:fe'})), record=qg-49c21fc0-4e) do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-01-24 17:01:11.624 55617 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] OVSDB transaction returned TRY_AGAIN, retrying do_commit /usr/lib/python2.7/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:93

Are the agents being run in containers?

Brian Haley (brian-haley) wrote :

The libipverbs.so error might be a red herring according to a downstream bug I found, but the others I'm not sure.

Changed in tripleo:
milestone: stein-3 → stein-rc1
wes hayutin (weshayutin) wrote :
Changed in tripleo:
status: Triaged → Fix Released
Changed in neutron:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers