tempest, access denied error in object storage tests failing in rocky build

Bug #1776729 reported by wes hayutin on 2018-06-13
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Quique Llorente

Bug Description

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/cd35af6/tempest.html.gz

Test Group/Test case Count Pass Fail Error Skip View
setUpClass (tempest.api.object_storage.test_account_quotas 1 0 1 0 0 Detail
setUpClass (tempest.api.object_storage.test_account_quotas_negative 1 0 1 0 0 Detail
setUpClass (tempest.api.object_storage.test_account_services 1 0 1 0 0 Detail
setUpClass (tempest.api.object_storage.test_container_staticweb 1 0 1 0 0 Detail
setUpClass (tempest.api.object_storage.test_container_sync 1 0 1 0 0 Detail
setUpClass (tempest.api.object_storage.test_container_sync_middleware 1 0 1 0 0 Detail
setUpClass (tempest.api.object_storage.test_object_expiry 1 0 1 0 0 Detail
setUpClass (tempest.api.object_storage.test_object_formpost 1 0 1 0 0 Detail
setUpClass (tempest.api.object_storage.test_object_formpost_negative 1 0 1 0 0 Detail
setUpClass (tempest.api.object_storage.test_object_temp_url 1 0 1 0 0 Detail
setUpClass (tempest.api.object_storage.test_object_temp_url_negative 1 0 1 0 0 Detail
tempest.api.object_storage.test_account_bulk.BulkTest 3 0 3 0 0 Detail
tempest.api.object_storage.test_container_acl.ObjectTestACLs 2 0 2 0 0 Detail
tempest.api.object_storage.test_container_acl_negative.ObjectACLsNegativeTest 9 0 9 0 0 Detail
tempest.api.object_storage.test_container_quotas.ContainerQuotasTest 3 0 3 0 0 Detail
tempest.api.object_storage.test_container_services.ContainerTest 24 0 24 0 0 Detail
tempest.api.object_storage.test_container_services_negative.ContainerNegativeTest 10 0 10 0 0 Detail
tempest.api.object_storage.test_object_slo.ObjectSloTest 4 0 4 0 0 Detail
tempest.api.object_storage.test_object_version.ContainerTest 1 0 1 0 0 Detail
tempest.scenario.test_object_storage_basic_ops.TestObjectStorageBasicOps 2 0 2 0 0 Detail
keystone_tempest_plugin.tests.api.identity.v3.test_identity_providers.IndentityProvidersTest 12 12 0 0 0 Detail

Bringing up a reproducer environment now.. Considering waiving these errors for a master promotion. Filling in the details now

wes hayutin (weshayutin) wrote :
Download full text (4.6 KiB)

Disk not found

https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-master/cd35af6/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.1.gz

2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager [req-0a85bc8b-5d49-4dd3-b513-70ab4eef7f14 4c02901af6cc4157978495477773ee3f 16a00adc86af4e049dd534472496748b - default default] Error updating resources for node overcloud-novacompute-0.localdomain.: DiskNotFound: No disk at /var/lib/nova/instances/a9b2420b-01b9-4c1c-b500-456b606fe850/disk
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager Traceback (most recent call last):
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7348, in _update_available_resource_for_node
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 670, in update_available_resource
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager resources = self.driver.get_available_resource(nodename)
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6436, in get_available_resource
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager disk_over_committed = self._get_disk_over_committed_size_total()
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8044, in _get_disk_over_committed_size_total
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager err_ctxt.reraise = False
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager self.force_reraise()
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb)
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova-compute.log.1: 7 ERROR nova.compute.manager File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 7990, in _get_disk_over_committed_size_total
2018-06-13 15:49:51.360 ERROR /var/log/containers/nova/nova...

Read more...

wes hayutin (weshayutin) wrote :

2018-06-13 15:49:33.110 ERROR /var/log/extra/docker/containers/logrotate_crond/log/containers/nova/nova-api.log.1: 17 ERROR nova.volume.cinder [req-a72b851e-0d63-4d18-81bd-a9b79f5be244 1faa45c9ca9b48d89d75053d881c784c 6f1aa8d97c1044809e146fe1a11bc569 - default default] [instance: 1d38a3ea-d416-4156-8f2d-3794e0018686] Create attachment failed for volume 5625929c-ef76-4e6f-946d-6664bd838160. Error: Invalid volume: Volume 5625929c-ef76-4e6f-946d-6664bd838160 status must be available or downloading (HTTP 400) (Request-ID: req-41b90300-7948-465f-bd52-c46a4a256efd) Code: 400: BadRequest: Invalid volume: Volume 5625929c-ef76-4e6f-946d-6664bd838160 status must be available or downloading (HTTP 400) (Request-ID: req-41b90300-7948-465f-bd52-c46a4a256efd)

wes hayutin (weshayutin) wrote :

Looks like the setup class is failing in most of these tests w/

ft2.1: setUpClass (tempest.api.object_storage.test_account_quotas_negative.AccountQuotasNegativeTest)_StringException: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 172, in setUpClass
    six.reraise(etype, value, trace)
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 165, in setUpClass
    cls.resource_setup()
  File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_account_quotas_negative.py", line 37, in resource_setup
    cls.create_container()
  File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/base.py", line 113, in create_container
    cls.container_client.update_container(container_name)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/object_storage/container_client.py", line 37, in update_container
    resp, body = self.put(url, body=None, headers=headers)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 343, in put
    return self.request('PUT', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 769, in _error_checker
    raise exceptions.Forbidden(resp_body, resp=resp)
tempest.lib.exceptions.Forbidden: Forbidden
Details: <html><h1>Forbidden</h1><p>Access was denied to this resource.</p></html>

summary: - tempest 69 object storage tests failing in rocky build
+ tempest, access denied error in object storage tests failing in rocky
+ build
Quique Llorente (quiquell) wrote :

Fix: ttps://review.openstack.org/#/c/575501/

Changed in tripleo:
assignee: nobody → Quique Llorente (quiquell)
status: Triaged → In Progress
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers