tempest.api.object_storage.test_object_services is failing on scenario002

Bug #1773325 reported by Arx Cruz on 2018-05-25
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Arx Cruz

Bug Description

Logs: http://logs.openstack.org/67/570167/1/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/18e0db5/logs/tempest.html.gz

Error:
ft1.1: setUpClass (tempest.api.object_storage.test_object_services.ObjectTest)_StringException: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 172, in setUpClass
    six.reraise(etype, value, trace)
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 165, in setUpClass
    cls.resource_setup()
  File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_object_services.py", line 36, in resource_setup
    cls.container_name = cls.create_container()
  File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/base.py", line 113, in create_container
    cls.container_client.update_container(container_name)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/object_storage/container_client.py", line 37, in update_container
    resp, body = self.put(url, body=None, headers=headers)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 343, in put
    return self.request('PUT', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 794, in _error_checker
    raise exceptions.PreconditionFailed(resp_body, resp=resp)
tempest.lib.exceptions.PreconditionFailed: Precondition Failed
Details: Bad URL

Still investigating, meanwhile, adding this test in the skip list

Reviewed: https://review.openstack.org/570533
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=f2eb646202f4d9ef52904cdf840dd7308b450479
Submitter: Zuul
Branch: master

commit f2eb646202f4d9ef52904cdf840dd7308b450479
Author: Arx Cruz <email address hidden>
Date: Fri May 25 10:32:07 2018 +0200

    Adding test_object_services to skip list

    This test is continuously failing on the gates in scenario002

    Change-Id: Ib0d47e7adc72f3a5f9387f87ef11c0ec78f72f06
    Related-Bug: #1773325

Matt Young (halcyondude) on 2018-06-04
tags: removed: quickstart

Reviewed: https://review.openstack.org/580423
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=7ee0e88208a07debd2839a41c56ee9570ed71d40
Submitter: Zuul
Branch: master

commit 7ee0e88208a07debd2839a41c56ee9570ed71d40
Author: Arx Cruz <email address hidden>
Date: Thu Jul 5 12:39:37 2018 -0300

    Removing test from skip list

    The test is no longer failing

    Change-Id: Ib784a47abf5419c188bb17cbe0aacde5a9b4344e
    Related-Bug: 1773325

wes hayutin (weshayutin) on 2018-07-12
Changed in tripleo:
milestone: none → rocky-3
tags: added: alert
removed: promotion-blocker
Ben Nemec (bnemec) wrote :

I take it this was not actually fixed, despite what the commit messages suggest? I know my patches are still failing on it pretty consistently.

wes hayutin (weshayutin) on 2018-07-13
tags: added: promotion-blocker

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/582339
Reason: scenario002 is failing in the gate, this patch needs to be out of the gate until we fix CI. I'll restore this patch in a few hours as the fix is actually in the gate. Please do not restore or recheck.

chandan kumar (chkumar246) wrote :
Download full text (18.1 KiB)

Some more logs from http://logs.openstack.org/18/581918/5/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/3cf2734/logs/undercloud/var/log/containers/tempest/tempest.log.txt.gz#_2018-07-13_04_47_43_593 giving 412 status code.

2018-07-13 04:47:43.582 81 INFO tempest.lib.common.rest_client [tx0059ce08a31c4dd4bd072-005b482eef ] Request (ObjectTest:setUpClass): 200 GET http://192.168.24.7:8080/info 0.431s
2018-07-13 04:47:43.583 81 DEBUG tempest.lib.common.rest_client [tx0059ce08a31c4dd4bd072-005b482eef ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '200', u'content-length': '1643', 'content-location': 'http://192.168.24.7:8080/info', u'connection': 'close', u'x-trans-id': 'tx0059ce08a31c4dd4bd072-005b482eef', u'date': 'Fri, 13 Jul 2018 04:47:43 GMT', u'content-type': 'application/json; charset=UTF-8', u'x-openstack-request-id': 'tx0059ce08a31c4dd4bd072-005b482eef'}
        Body: {"formpost": {}, "bulk_delete": {"max_failed_deletes": 1000, "max_deletes_per_request": 10000}, "versioned_writes": {"allowed_flags": ["x-versions-location", "x-history-location"]}, "bulk_upload": {"max_failed_extractions": 1000, "max_containers_per_extraction": 10000}, "container_quotas": {}, "tempurl": {"outgoing_remove_headers": ["x-object-meta-*"], "methods": ["GET", "HEAD", "PUT", "POST", "DELETE"], "incoming_remove_headers": ["x-timestamp"], "allowed_digests": ["sha1", "sha256", "sha512"], "outgoing_allow_headers": ["x-object-meta-public-*"], "incoming_allow_headers": []}, "s3api": {"max_upload_part_num": 1000, "allow_multipart_uploads": true, "max_bucket_listing": 1000, "s3_acl": false, "min_segment_size": 5242880, "max_parts_listing": 1000, "max_multi_delete_objects": 1000}, "ratelimit": {"account_ratelimit": 0.0, "max_sleep_time_seconds": 60.0, "container_ratelimits": [], "container_listing_ratelimits": []}, "slo": {"max_manifest_segments": 1000, "yield_frequency": 10, "min_segment_size": 1, "max_manifest_size": 2097152}, "account_quotas": {}, "staticweb": {}, "swift": {"max_file_size": 5368709122, "account_listing_limit": 10000, "account_autocreate": true, "max_meta_count": 90, "max_meta_value_length": 256, "container_listing_limit": 10000, "extra_header_count": 0, "max_meta_overall_size": 4096, "version": "2.18.1.dev169", "max_meta_name_length": 128, "max_header_size": 8192, "policies": [{"default": true, "name": "Policy-0", "aliases": "Policy-0"}], "max_object_name_length": 1024, "max_account_name_length": 256, "strict_cors_mode": true, "allow_account_management": true, "max_container_name_length": 256}} _log_request_full /usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py:434
2018-07-13 04:47:43.592 81 INFO tempest.lib.common.rest_client [txb87cc715719a4d748bdbd-005b482eef ] Request (ObjectTest:setUpClass): 412 PUT http://192.168.24.7:8080/tempest-TestContainer-1152572772 0.008s
2018-07-13 04:47:43.592 81 DEBUG tempest.lib.common.rest_client [txb87cc715719a4d748bdbd-005b482eef ] Request - Headers: {'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '412', u'content-length': '...

Reviewed: https://review.openstack.org/582339
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=6b05d5ab400bda8377eed3d524a5b354dde79ff5
Submitter: Zuul
Branch: master

commit 6b05d5ab400bda8377eed3d524a5b354dde79ff5
Author: Alex Schultz <email address hidden>
Date: Thu Jul 12 19:00:04 2018 +0000

    Revert "Removing test from skip list"

    This reverts commit 7ee0e88208a07debd2839a41c56ee9570ed71d40.

    It's still failing

    Change-Id: Icd17453092a0c1043b46ff7307f8d48e0e72eb63
    Related-Bug: 1773325

Christian Schwede (cschwede) wrote :

Thanks Chandan, now it is more clear why Swift complains.

2018-07-13 04:47:43.592 81 INFO tempest.lib.common.rest_client [txb87cc715719a4d748bdbd-005b482eef ] Request (ObjectTest:setUpClass): 412 PUT http://192.168.24.7:8080/tempest-TestContainer-1152572772 0.008s

The request uses http://192.168.24.7:8080/tempest-TestContainer-1152572772 as URL, and Swift doesn't know how to handle this.

It should look sth like this: http://192.168.24.7:8080/v1/AUTH_<tenant UUID>/tempest-TestContainer-1152572772

So the "v1/AUTH_<tenant UUID>" part is missing from the Tempest-generated URL. I'm not sure where this URL is generated in Tempest, but I think this is not a Swift issue, but a Tempest one.

Arx Cruz (arxcruz) wrote :

Tempest get these urls from the endpoint publicURL in openstack

wes hayutin (weshayutin) wrote :

@arx @Christian so how to proceed? Arx can you please post a paste of the publicURL printed out by the swift endpoint.

Christian I suppose there are two ways to fix this..
1. hardcode the url path in python-tempestconf
2. fix what is return by endpoint list from keystone.

WDYT?

wes hayutin (weshayutin) wrote :

Is this related at all ?
http://logs.openstack.org/37/579237/1/gate/tripleo-ci-centos-7-undercloud-containers/959e942/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-07-13_20_04_15

Creating a new temporary URL for project service",
2018-07-13 20:04:15 | "stderr: Unable to establish connection to https://192.168.24.2:13808/v1/AUTH_b4a6c3a1aeaf4a90b1fab78bdd523264: ('Connection aborted.', BadStatusLine(\"''\",))",

Fix proposed to branch: master
Review: https://review.openstack.org/583659

Changed in tripleo:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/583659
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=018d057f49bcac509ca67b56b4075a0e44209254
Submitter: Zuul
Branch: master

commit 018d057f49bcac509ca67b56b4075a0e44209254
Author: Arx Cruz <email address hidden>
Date: Wed Jul 18 13:22:53 2018 -0300

    Fix tempest-deployer-input.conf path

    Tempest containerized run as root, and so, although the
    tempest-deployer-input.conf file was being imported to the container,
    the patch was ~/tempest-deployer-input.conf that was pointing to /root
    instead of working_dir, so, tempestconf wasn't getting the defaults from
    the deployer input file, setting the region to RegionOne instead of
    regionOne, and so, the object_store tests were failing, because was
    getting the default endpoint http://.../ instead of http://.../v1/AUTH

    Change-Id: Ic88c9121441b4021da3400933cacc4ff942b727f
    Closes-Bug: 1773325

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers