tripleo

[fedora28][check] tempest boot volume pattern and network basic ops tests are failing

Bug #1830718 reported by chandan kumar on 2019-05-28

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Won't Fix	Critical	Unassigned	tripleo train-3

Bug Description

tripleo-ci-fedora-28-standalone job are failing consistently at following tests:
* tempest.scenario.test_network_basic_ops.TestNetworkBasicOps [1.]
* tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern [2.]

Both are failing with same reason:
[1.] http://logs.openstack.org/97/661697/1/check/tripleo-ci-fedora-28-standalone/e55278e/logs/tempest.html.gz
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_basic_ops.py", line 708, in test_preserve_preexisting_port
    self._setup_network_and_servers(boot_with_port=True)
  File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_basic_ops.py", line 119, in _setup_network_and_servers
    server = self._create_server(self.network, port_id)
  File "/usr/lib/python3.6/site-packages/tempest/scenario/test_network_basic_ops.py", line 171, in _create_server
    security_groups=security_groups)
  File "/usr/lib/python3.6/site-packages/tempest/scenario/manager.py", line 235, in create_server
    image_id=image_id, **kwargs)
  File "/usr/lib/python3.6/site-packages/tempest/common/compute.py", line 265, in create_test_server
    server['id'])
  File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/tempest/common/compute.py", line 236, in create_test_server
    clients.servers_client, server['id'], wait_until)
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server 728e4226-3ffa-4a34-b570-8f84601a72fb failed to build and is in ERROR status
Details: {'code': 500, 'created': '2019-05-28T10:03:57Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 728e4226-3ffa-4a34-b570-8f84601a72fb.'}

[2.] http://logs.openstack.org/99/661599/1/gate/tripleo-ci-fedora-28-standalone/6a1da25/logs/tempest.html.gz

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 89, in wrapper
    return f(*func_args, **func_kwargs)
  File "/usr/lib/python3.6/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 131, in test_volume_boot_pattern
    security_group=security_group)
  File "/usr/lib/python3.6/site-packages/tempest/scenario/test_volume_boot_pattern.py", line 68, in _boot_instance_from_resource
    return self.create_server(image_id='', **create_kwargs)
  File "/usr/lib/python3.6/site-packages/tempest/scenario/manager.py", line 235, in create_server
    image_id=image_id, **kwargs)
  File "/usr/lib/python3.6/site-packages/tempest/common/compute.py", line 265, in create_test_server
    server['id'])
  File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3.6/site-packages/tempest/common/compute.py", line 236, in create_test_server
    clients.servers_client, server['id'], wait_until)
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server 6312266c-263b-4ca5-8484-85f019af7db2 failed to build and is in ERROR status
Details: {'code': 500, 'created': '2019-05-28T08:34:25Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 6312266c-263b-4ca5-8484-85f019af7db2.'}

Needs to investigate further to find the failure reasons.

Tags:

Revision history for this message

Arx Cruz (arxcruz) wrote on 2019-05-28:

Download full text (9.0 KiB)

http://logs.openstack.org/99/661599/1/gate/tripleo-ci-fedora-28-standalone/6a1da25/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz#_2019-05-28_08_34_15_142

This seems to be related to concurrency on network, we should have network guys taking a look.

http://logs.openstack.org/99/661599/1/gate/tripleo-ci-fedora-28-standalone/6a1da25/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz#_2019-05-28_08_34_15_142

This seems to be related to concurrency on network, we should have network guys taking a look.

2019-05-28 08:34:15.142 8 ERROR vif_plug_ovs.ovsdb.impl_vsctl [req-e7b4f0d3-33d1-4dee-8646-de4ac02b6149 f316983829a74e1eaf29f164c4e5cecc 246b9cbfe0a94514962f71bab62a30f4 - default default] Unable to execute ['ovs-vsctl', '--timeout=120', '--oneline', '--format=json', '--db=tcp:127.0.0.1:6640', '--', '--may-exist', 'add-br', 'br-int', '--', 'set', 'Bridge', 'br-int', 'datapath_type=system']. Exception: Unexpected error while running command.
Command: ovs-vsctl --timeout=120 --oneline --format=json --db=tcp:127.0.0.1:6640 -- --may-exist add-br br-int -- set Bridge br-int datapath_type=system
Exit code: 1
Stdout: ''
Stderr: 'ovs-vsctl: tcp:127.0.0.1:6640: database connection failed (Connection refused)\n': oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2019-05-28 08:34:15.143 8 INFO os_vif [req-e7b4f0d3-33d1-4dee-8646-de4ac02b6149 f316983829a74e1eaf29f164c4e5cecc 246b9cbfe0a94514962f71bab62a30f4 - default default] Successfully plugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:0b:35:f8,bridge_name='br-int',has_traffic_filtering=True,id=3926672a-c06d-4455-a7c2-a718c0d15907,network=Network(1dfcc0ad-a672-43d5-800b-52d4413307a8),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_delete=False,vif_name='tap3926672a-c0')
2019-05-28 08:34:15.287 8 DEBUG nova.virt.libvirt.driver [req-e7b4f0d3-33d1-4dee-8646-de4ac02b6149 f316983829a74e1eaf29f164c4e5cecc 246b9cbfe0a94514962f71bab62a30f4 - default default] No BDM found with device name vda, not building metadata. _build_disk_metadata /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:9366
2019-05-28 08:34:15.288 8 DEBUG nova.virt.libvirt.driver [req-e7b4f0d3-33d1-4dee-8646-de4ac02b6149 f316983829a74e1eaf29f164c4e5cecc 246b9cbfe0a94514962f71bab62a30f4 - default default] No VIF found with MAC fa:16:3e:0b:35:f8, not building metadata _build_interface_metadata /usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py:9342
2019-05-28 08:34:15.377 8 DEBUG nova.network.neutronv2.api [req-7051defb-3e74-4dd8-adb3-c03c211f6724 307ac996d216442ab5a5659f4b0c0aec 3a5aed9cb1b64566bf5646998fbba091 - default default] [instance: 6312266c-263b-4ca5-8484-85f019af7db2] Updated VIF entry in instance network info cache for port 3926672a-c06d-4455-a7c2-a718c0d15907. _build_network_info_model /usr/lib/python3.6/site-packages/nova/network/neutronv2/api.py:2971
2019-05-28 08:34:15.378 8 DEBUG nova.network.base_api [req-7051defb-3e74-4dd8-adb3-c03c211f6724 307ac996d216442ab5a5659f4b0c0aec 3a5aed9cb1b64566bf5646998fbba091 - default default] [instance: 6312266c-263b-4ca5-8484-85f019af7db2] Updating instance_info_cache with network_info: [{"id": "3926672a-c06d-4455-a7c2-a718c0d15907", "address": "fa:16:3e:0b:35:f8", "network": {"id": "1dfcc0ad-a672-43d5-800b-52d4413307a8", "bridge": "br-int", "label": "tempest-TestVolumeBootPattern-1415094620-network", "subnets": [{"cidr": "10.100.0.0/28", "dns": [], "gateway": {"address": "10.100.0.1", "type": "gateway", "version": 4, "meta": {}}, "ips": [{"address": "10.100.0.11", "type": "fixed", "version": 4, "meta": {}, "floating_ips": []}], "routes": [], "version": 4, "meta": {"dhcp_server": "10.100.0.2"}}], "meta": {"injected": false, "tenant_id": "246b9cbfe0a94514962f71bab62a30f4", "mtu": 1442, "physical_network": null, "tunneled": true}}, "type": "ovs", "details": {"port_filter": true}, "devname": "tap3926672a-c0", "ovs_interfaceid": "3926672a-c06d-4455-a7c2-a718c0d15907", "qbh_params": null, "qbg_params": null, "active": false, "vnic_type": "normal", "profile": {}, "preserve_on_delete": false, "meta": {}}] update_instance_cache_with_nw_info /usr/lib/python3.6/site-packages/nova/network/base_api.py:47
2019-05-28 08:34:15.415 8 DEBUG oslo_concurrency.lockutils [req-7051defb-3e74-4dd8-adb3-c03c211f6724 307ac996d216442ab5a5659f4b0c0aec 3a5aed9cb1b64566bf5646998fbba091 - default default] Releasing lock "refresh_cache-6312266c-263b-4ca5-8484-85f019af7db2" lock /usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py:281
2019-05-28 08:34:16.321 8 ERROR nova.virt.libvirt.guest [req-e7b4f0d3-33d1-4dee-8646-de4ac02b6149 f316983829a74e1eaf29f164c4e5cecc 246b9cbfe0a94514962f71bab62a30f4 - default default] Error launching a defined domain with XML: <domain type='qemu'>
  <name>instance-00000018</name>
  <uuid>6312266c-263b-4ca5-8484-85f019af7db2</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="19.1.0-0.20190525010931.37ccd7e.fc28"/>
      <nova:name>tempest-TestVolumeBootPattern-server-2102477827</nova:name>
      <nova:creationTime>2019-05-28 08:34:13</nova:creationTime>
      <nova:flavor name="m1.nano">
        <nova:memory>64</nova:memory>
        <nova:disk>1</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>1</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="f316983829a74e1eaf29f164c4e5cecc">tempest-TestVolumeBootPattern-1902746153</nova:user>
        <nova:project uuid="246b9cbfe0a94514962f71bab62a30f4">tempest-TestVolumeBootPattern-1902746153</nova:project>
      </nova:owner>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>65536</memory>
  <currentMemory unit='KiB'>65536</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <cputune>
    <shares>1024</shares>
  </cputune>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>RDO</entry>
      <entry name='product'>OpenStack Compute</entry>
      <entry name='version'>19.1.0-0.20190525010931.37ccd7e.fc28</entry>
      <entry name='serial'>6312266c-263b-4ca5-8484-85f019af7db2</entry>
      <entry name='uuid'>6312266c-263b-4ca5-8484-85f019af7db2</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.11'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sda'/>
      <target dev='vda' bus='virtio'/>
      <serial>ce4323ab-802c-4109-9970-72f4b6817013</serial>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='fa:16:3e:0b:35:f8'/>
      <source bridge='br-int'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='3926672a-c06d-4455-a7c2-a718c0d15907'/>
      </virtualport>
      <target dev='tap3926672a-c0'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <mtu size='1442'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <log file='/var/lib/nova/instances/6312266c-263b-4ca5-8484-85f019af7db2/console.log' append='off'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <log file='/var/lib/nova/instances/6312266c-263b-4ca5-8484-85f019af7db2/console.log' append='off'/>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='192.168.24.1'>
      <listen type='address' address='192.168.24.1'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <stats period='10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </memballoon>
    <rng model='virtio'>
      <backend model='random'>/dev/urandom</backend>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </rng>
  </devices>
</domain>
: libvirt.libvirtError: internal error: Unable to add port tap3926672a-c0 to OVS bridge br-int
2019-05-28 08:34:16.322 8 ERROR nova.virt.libvirt.driver [req-e7b4f0d3-33d1-4dee-8646-de4ac02b6149 f316983829a74e1eaf29f164c4e5cecc 246b9cbfe0a94514962f71bab62a30f4 - default default] [instance: 6312266c-263b-4ca5-8484-85f019af7db2] Failed to start libvirt guest: libvirt.libvirtError: internal error: Unable to add port tap3926672a-c0 to OVS bridge br-int

Alex Schultz (alex-schultz) on 2019-06-07

Changed in tripleo:
milestone:	train-1 → train-2

Revision history for this message

chandan kumar (chkumar246) wrote on 2019-06-10:

http://zuul.openstack.org/builds?job_name=tripleo-ci-fedora-28-standalone appears to be quite flakky, we need to keey an eye on it.

Alex Schultz (alex-schultz) on 2019-07-29

Changed in tripleo:
milestone:	train-2 → train-3

Revision history for this message

chandan kumar (chkumar246) wrote on 2019-08-20:

Closing as Fedora 28 is replaced with RHEL-8

Changed in tripleo:
status:	Triaged → Won't Fix

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.