tripleo

[master][promotion]tempest.scenario.test_network_basic_ops.TestNetworkBasicOps tests failed in FS18 and FS19

Bug #1782095 reported by chandan kumar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Triaged
Critical
Sagi (Sergey) Shnaidman
tripleo rocky-3

Bug Description

on master promotion pipeline, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps tests failed in FS18 and FS19.
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-master/ba7e98c/logs/tempest.html.gz

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_network_basic_ops.py", line 406, in test_network_basic_ops
    self._setup_network_and_servers()
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_network_basic_ops.py", line 119, in _setup_network_and_servers
    server = self._create_server(self.network, port_id)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_network_basic_ops.py", line 171, in _create_server
    security_groups=security_groups)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 210, in create_server
    image_id=image_id, **kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/common/compute.py", line 258, in create_test_server
    server['id'])
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/tempest/common/compute.py", line 229, in create_test_server
    clients.servers_client, server['id'], wait_until)
  File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server a5f3222d-61a7-43a6-8e8f-ad8ddba9fc93 failed to build and is in ERROR status
Details: {u'message': u'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance a5f3222d-61a7-43a6-8e8f-ad8ddba9fc93.', u'code': 500, u'created': u'2018-07-17T01:49:31Z'}

https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/5963ae2/logs/tempest.html.gz

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_network_basic_ops.py", line 406, in test_network_basic_ops
    self._setup_network_and_servers()
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_network_basic_ops.py", line 119, in _setup_network_and_servers
    server = self._create_server(self.network, port_id)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/test_network_basic_ops.py", line 171, in _create_server
    security_groups=security_groups)
  File "/usr/lib/python2.7/site-packages/tempest/scenario/manager.py", line 210, in create_server
    image_id=image_id, **kwargs)
  File "/usr/lib/python2.7/site-packages/tempest/common/compute.py", line 258, in create_test_server
    server['id'])
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/tempest/common/compute.py", line 229, in create_test_server
    clients.servers_client, server['id'], wait_until)
  File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 78, in wait_for_server_status
    raise exceptions.BuildErrorException(server_id=server_id)
tempest.exceptions.BuildErrorException: Server 5a005df3-e601-4334-adfd-ce5182232129 failed to build and is in ERROR status

More info is here https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/5963ae2/logs/subnode-2/var/log/extra/errors.txt.gz#_2018-07-17_01_53_36_092

2018-07-17 01:53:36.092 ERROR /var/log/extra/docker/containers/nova_api_cron/log/nova/nova-conductor.log: 8 ERROR nova.scheduler.utils [req-69e5c131-3e75-4c8e-a59b-1ecad92e4163 3e3fd98f92e0419aa3cf86d9068c629c feda49c2c44949a08ed943bb75b76222 - default default] [instance: 5a005df3-e601-4334-adfd-ce5182232129] Error from last host: upstream-centos-7-rdo-cloud-tripleo-0000061569.localdomain (node upstream-centos-7-rdo-cloud-tripleo-0000061569.localdomain): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1832, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2121, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 5a005df3-e601-4334-adfd-ce5182232129 was re-scheduled: Binding failed for port 6eadb9d1-661c-4804-a075-35175ac17a99, please check neutron logs for more information.\n']

Revision history for this message
chandan kumar (chkumar246) wrote :

based on discussion with slaweq, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/5963ae2/logs/subnode-2/var/log/containers/neutron/server.log.txt.gz#_2018-07-17_01_53_34_039

2018-07-17 01:53:34.065 29 ERROR neutron.plugins.ml2.managers [req-24f22378-e6fc-4bc2-accf-a71f98b3ab3d e7523f3e2d6844c7ae9c484705fdfeac 155b861a8dd14991858bd08631c48fd1 - default default] Failed to bind port 6eadb9d1-661c-4804-a075-35175ac17a99 on host upstream-centos-7-rdo-cloud-tripleo-0000061569.localdomain for vnic_type normal using segments [{'network_id': 'c6003379-48b6-4161-b9dc-55014daca65c', 'segmentation_id': 17, 'physical_network': None, 'id': 'f48a8f56-1ce3-408d-8fbf-f10480e06b43', 'network_type': u'vxlan'}]

There is no OVS agent which can bind this port and the agent was not started, we need to look for OVS agent log.

Revision history for this message
chandan kumar (chkumar246) wrote :

 amoralej │ Jul 17 01:49:08 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: ImportError: cannot import name utils
 amoralej │ https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr
          │ -featureset019-master/5963ae2/logs/subnode-2/var/log/journal.txt.gz
 amoralej │ that's why ovs agent is not starting

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

Is should be fixed by https://review.rdoproject.org/r/#/c/14859

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

OVS logs are there:
https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/legacy-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/5963ae2/logs/subnode-2/var/log/openvswitch/

Jul 17 01:56:28 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: time="2018-07-17T01:56:28.224665868Z" level=warning msg="a3e76b7253b2974c65c426a182fbc77ba50f6905b2b2b27149b4f645cd142942 unmountVolumes: Failed to do lazy umount fo volume '/var/lib/docker/overlay2/f389d8fdd7950957ae002258bbc1a93b85982a301826c38eccb117846b3a780b/merged/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt': invalid argument"
Jul 17 01:56:28 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: time="2018-07-17T01:56:28.244541758Z" level=warning msg="a3e76b7253b2974c65c426a182fbc77ba50f6905b2b2b27149b4f645cd142942 unmountVolumes: Failed to do lazy umount fo volume '/var/lib/docker/overlay2/f389d8fdd7950957ae002258bbc1a93b85982a301826c38eccb117846b3a780b/merged/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem': invalid argument"
Jul 17 01:56:28 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: time="2018-07-17T01:56:28.309506328Z" level=warning msg="a3e76b7253b2974c65c426a182fbc77ba50f6905b2b2b27149b4f645cd142942 unmountVolumes: Failed to do lazy umount fo volume '/var/lib/docker/overlay2/f389d8fdd7950957ae002258bbc1a93b85982a301826c38eccb117846b3a780b/merged/etc/hosts': invalid argument"
Jul 17 01:56:28 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: time="2018-07-17T01:56:28.320222637Z" level=error msg="Error setting up exec command in container neutron_ovs_agent: Container a3e76b7253b2974c65c426a182fbc77ba50f6905b2b2b27149b4f645cd142942 is restarting, wait until the container is running"

Jul 17 01:49:46 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: Traceback (most recent call last):
Jul 17 01:49:46 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
Jul 17 01:49:46 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: "__main__", fname, loader, pkg_name)
Jul 17 01:49:46 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
Jul 17 01:49:46 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: exec code in run_globals
Jul 17 01:49:46 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: File "/usr/lib/python2.7/site-packages/neutron/cmd/destroy_patch_ports.py", line 25, in <module>
Jul 17 01:49:46 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: from neutron.plugins.common import utils as p_utils
Jul 17 01:49:46 upstream-centos-7-rdo-cloud-tripleo-0000061569 dockerd-current[2936]: ImportError: cannot import name utils

Seems like it's the same problem as in https://bugs.launchpad.net/tripleo/+bug/1782088

Changed in tripleo:
assignee: nobody → Sagi (Sergey) Shnaidman (sshnaidm)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.