OpenStack Compute (nova)

nova-lvm job constantly fails on Glance image upload

Bug #1938765 reported by Sylvain Bauza
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Critical
Unassigned

Bug Description

The nova-lvm job seems to fail for every change [1] which prevents us to merge any change touching nova/virt/libvirt [2]

All failures seem to relate to the same Tempest tests (6 of them) failing with the same problem : a Glance image upload issue as Glance API returns a HTTP502.

Aug 01 03:50:12.036077 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR os_brick.initiator.linuxscsi [None req-8fa95fb0-15e8-4bf7-8314-0d2dac1c9b9c tempest-VolumesAdminNegativeTest-243853426 tempest-VolumesAdminNegativeTest-243853426-project] multipathd is not running: exit code None: oslo_concurrency.processutils.ProcessExecutionError: [Errno 2] No such file or directory

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server [None req-76a536ab-394f-4347-8dec-30842c1ec1d2 tempest-ListImageFiltersTestJSON-1499681503 tempest-ListImageFiltersTestJSON-1499681503-project] Exception during message handling: glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2916, in snapshot

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server metadata['location'] = root_disk.direct_snapshot(

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 452, in direct_snapshot

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server raise NotImplementedError(_('direct_snapshot() is not implemented'))

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not implemented

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 71, in wrapped

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server _emit_versioned_exception_notification(

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server self.force_reraise()

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server raise self.value

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/exception_wrapper.py", line 63, in wrapped

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 183, in decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. "

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server self.force_reraise()

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server raise self.value

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 154, in decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/utils.py", line 1433, in decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 211, in decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context,

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server self.force_reraise()

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server raise self.value

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 200, in decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 231, in decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server compute_utils.delete_image(

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server self.force_reraise()

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server raise self.value

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 227, in decorated_function

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server return function(self, context, image_id, instance,

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 3917, in snapshot_instance

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server self._snapshot_instance(context, image_id, instance,

Aug 01 03:51:41.348545 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/compute/manager.py", line 3950, in _snapshot_instance

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server self.driver.snapshot(context, instance, image_id,

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2987, in snapshot

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server self._image_api.update(context,

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 1240, in update

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server return session.update(context, image_id, image_info, data=data,

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 690, in update

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 1028, in _reraise_translated_image_exception

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 688, in update

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server image = self._update_v2(context, sent_service_image_meta, data)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 707, in _update_v2

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server image = self._upload_data(context, image_id, data)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 583, in _upload_data

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server utils.tpool_execute(self._client.call,

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/utils.py", line 687, in tpool_execute

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server eventlet.tpool.execute(func, *args, **kwargs)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 132, in execute

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server raise value

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/eventlet/tpool.py", line 86, in tworker

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/opt/stack/nova/nova/image/glance.py", line 190, in call

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server result = getattr(controller, method)(*args, **kwargs)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/glanceclient/common/utils.py", line 628, in inner

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/glanceclient/v2/images.py", line 308, in upload

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server resp, body = self.http_client.put(url, headers=hdrs, data=body)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/keystoneauth1/adapter.py", line 404, in put

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py", line 380, in request

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server return self._handle_response(resp)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/glanceclient/common/http.py", line 120, in _handle_response

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server raise exc.from_response(resp, resp.content)

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPBadGateway: HTTP 502 Bad Gateway: Bad Gateway: The proxy server received an invalid: response from an upstream server.: Apache/2.4.41 (Ubuntu) Server at 158.69.72.121 Port 80

Aug 01 03:51:41.355434 ubuntu-focal-ovh-bhs1-0025715861 nova-compute[106038]: ERROR oslo_messaging.rpc.server

[1] https://zuul.openstack.org/builds?job_name=nova-lvm&branch=master
[2] https://zuul.openstack.org/job/nova-lvm

Tags: gate-failure
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/803322

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.opendev.org/c/openstack/nova/+/803322
Committed: https://opendev.org/openstack/nova/commit/dbf3e276ba5971f1f6483e04b84cb0fc403cd592
Submitter: "Zuul (22348)"
Branch: master

commit dbf3e276ba5971f1f6483e04b84cb0fc403cd592
Author: Lee Yarwood <email address hidden>
Date: Tue Aug 3 10:05:42 2021 +0100

    zuul: Increase GLANCE_LIMIT_IMAGE_SIZE_TOTAL for nova-lvm

    As seen before in bug #1913451 the nova-lvm job creates large RAW
    snapshots during each run due to the nature of using LVM storage for the
    ephemeral instance disks.

    Id425aa546f1a5973bae8be9c017782d18f0b4a47 recently landed capping the
    per teneant image size total to 1000 MiB that is far too low for this
    job given the RAW snapshots it's creating. This in turn resulted in
    504 failures when n-cpu attempted to upload snapshots to g-api.

    This change simply increases the per tenant limit to 10 GiB that should
    be more than enough for all of the tests covered by the nova-lvm job.

    Closes-Bug: #1938765
    Change-Id: I824655387a10ac9b813c1b7b2399e25ed95f7fc3

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 24.0.0.0rc1

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers