CI: neutron.tests.tempest.scenario.test_qos.QoSTest.test_qos fails in pike promotion jobs

Bug #1737940 reported by Sagi (Sergey) Shnaidman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
John Trowbridge

Bug Description

neutron.tests.tempest.scenario.test_qos.QoSTest.test_qos fails in pike periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike job which run all tempest tests:

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/f8dc459/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-12-13_02_54_48

2017-12-13 02:54:48 | ~~~~~~~~~~~~~~~~~~~
2017-12-13 02:54:48 | Traceback (most recent call last):
2017-12-13 02:54:48 | File "/usr/lib/python2.7/site-packages/neutron/tests/tempest/scenario/test_qos.py", line 175, in test_qos
2017-12-13 02:54:48 | sleep=1)
2017-12-13 02:54:48 | File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 682, in wait_until_true
2017-12-13 02:54:48 | while not predicate():
2017-12-13 02:54:48 | File "/usr/lib/python2.7/site-packages/neutron/tests/tempest/scenario/test_qos.py", line 173, in <lambda>
2017-12-13 02:54:48 | port=NC_PORT),
2017-12-13 02:54:48 | File "/usr/lib/python2.7/site-packages/neutron/tests/tempest/scenario/test_qos.py", line 114, in _check_bw
2017-12-13 02:54:48 | data = client_socket.recv(QoSTest.BUFFER_SIZE)
2017-12-13 02:54:48 | File "/usr/lib/python2.7/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
2017-12-13 02:54:48 | raise TimeoutException()
2017-12-13 02:54:48 | fixtures._fixtures.timeout.TimeoutException

another run:

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/95cde78/undercloud/home/jenkins/tempest_output.log.txt.gz#_2017-12-13_06_39_04

There are failures in compute node:
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/f8dc459/overcloud-novacompute-0/var/log/extra/errors.txt.gz
/var/log/nova/nova-compute.log: 19417 ERROR nova.compute.manager
InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on /var/lib/nova/instances/18816472-b392-4a78-b2c8-577cfe106c92/disk : Unexpected error while running command.
2017-12-13 02:33:10.069 ERROR /var/log/nova/nova-compute.log: 19417 ERROR nova.compute.manager Command: /usr/bin/python2 -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info /var/lib/nova/instances/18816472-b392-4a78-b2c8-577cfe106c92/disk
2017-12-13 02:33:10.069 ERROR /var/log/nova/nova-compute.log: 19417 ERROR nova.compute.manager Exit code: 1
2017-12-13 02:33:10.069 ERROR /var/log/nova/nova-compute.log: 19417 ERROR nova.compute.manager Stdout: u''
2017-12-13 02:33:10.069 ERROR /var/log/nova/nova-compute.log: 19417 ERROR nova.compute.manager Stderr: u"qemu-img: Could not open '/var/lib/nova/instances/18816472-b392-4a78-b2c8-577cfe106c92/disk': Could not open '/var/lib/nova/instances/18816472-b392-4a78-b2c8-577cfe106c92/disk': No such file or directory\n"

On controller node neutron failures:
https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/f8dc459/overcloud-controller-0/var/log/extra/errors.txt.gz

ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: bind: Cannot assign requested address
2017-12-13 02:00:17.011 ERROR /var/log/neutron/l3-agent.log: 36275 ERROR neutron.agent.linux.ip_lib
2017-12-13 02:00:17.011 ERROR /var/log/neutron/l3-agent.log: 36275 ERROR neutron.agent.linux.ip_lib
2017-12-13 02:00:17.036 ERROR /var/log/neutron/l3-agent.log: 36275 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: bind: Cannot assign requested address
2017-12-13 02:00:17.060 ERROR /var/log/neutron/l3-agent.log: 36275 ERROR neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 192.168.24.101 on qg-12e4422c-8e in namespace qrouter-3c49d610-6e26-4208-a4f9-bc7a72e2c51b: Exit code: 2; Stdin: ; Stdout: ; Stderr: bind: Cannot assign requested address

Revision history for this message
Jakub Libosvar (libosvar) wrote :

neutron.tests.tempest.scenario.* tests are not even voting on Neutron gates because of instability. It shouldn't be run in promotion jobs.

Changed in tripleo:
assignee: nobody → chandan kumar (chkumar246)
status: Triaged → In Progress
Changed in tripleo:
assignee: chandan kumar (chkumar246) → John Trowbridge (trown)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.openstack.org/528030
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=154177a4fcd202fb038d654a9a3731d1741cdf37
Submitter: Zuul
Branch: master

commit 154177a4fcd202fb038d654a9a3731d1741cdf37
Author: Chandan Kumar <email address hidden>
Date: Thu Dec 14 23:54:08 2017 +0530

    Add neutron_tempest_plugin.scenario to skip list

    * neutron_tempest_plugin.scenario tests are not yet stable.
      In upstream CI, it is non-voting, so adding it to skip list.

    Closes-Bug: #1737940
    Change-Id: I9fb47520a5c74df2379695007acea77020c1af5a

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart-extras (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/635152

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.openstack.org/635152
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=fe62dcc603e19c4776d86e803ab463ace371801c
Submitter: Zuul
Branch: master

commit fe62dcc603e19c4776d86e803ab463ace371801c
Author: Brian Haley <email address hidden>
Date: Wed Feb 6 15:19:55 2019 +0200

    Re-enable neutron-tempest-plugin scenario tests

    These tests should be stable by now, let's re-enable them.

    Change-Id: Ieb99209e89e8b0c0cf26783cbf4e5fe72157600b
    Related-bug: #1737940

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-quickstart-extras 2.1.1

This issue was fixed in the openstack/tripleo-quickstart-extras 2.1.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart-extras (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/699558

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-quickstart-extras (master)

Change abandoned by Suraj Patil (<email address hidden>) on branch: master
Review: https://review.opendev.org/699558

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart-extras (master)

Reviewed: https://review.opendev.org/699394
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=f1b00f047569913eb33fe67c6cf714ca0e8b4373
Submitter: Zuul
Branch: master

commit f1b00f047569913eb33fe67c6cf714ca0e8b4373
Author: Soniya Vyas <email address hidden>
Date: Fri Jan 3 15:40:56 2020 +0530

    [Rocky] Removed passing tests from skiplist

    Class neutron_tempest_plugin.scenario has maximum
    number of tests passed. Hence, it is removed and
    only failing testsis kept.

    In addition to above, added two more tests from
    class 'TestSecurityGroupsBasicOps' and a test from
    class 'ServersOnMultiNodesTest'

    Related-bug: #1737940
    Related-bug: #1753209
    Related-bug: #1843259
    Related-bug: #1793482
    Related-bug: #1831223
    Related-bug: #1857365

    Signed-off by: Soniya Vyas<email address hidden>
    Change-Id: I8239bb694187d7f912163742e836a7362cdb1483

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on tripleo-quickstart-extras (master)

Change abandoned by "Brent Eagles <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/701341
Reason: 2 years old and applies to queens - probably safe to abandon

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.