tempest.api.object_storage.test_object_services is failing on scenario002

Bug #1773325 reported by Arx Cruz on 2018-05-25
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
chandan kumar

Bug Description

Logs: http://logs.openstack.org/67/570167/1/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/18e0db5/logs/tempest.html.gz

Error:
ft1.1: setUpClass (tempest.api.object_storage.test_object_services.ObjectTest)_StringException: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 172, in setUpClass
    six.reraise(etype, value, trace)
  File "/usr/lib/python2.7/site-packages/tempest/test.py", line 165, in setUpClass
    cls.resource_setup()
  File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/test_object_services.py", line 36, in resource_setup
    cls.container_name = cls.create_container()
  File "/usr/lib/python2.7/site-packages/tempest/api/object_storage/base.py", line 113, in create_container
    cls.container_client.update_container(container_name)
  File "/usr/lib/python2.7/site-packages/tempest/lib/services/object_storage/container_client.py", line 37, in update_container
    resp, body = self.put(url, body=None, headers=headers)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 343, in put
    return self.request('PUT', url, extra_headers, headers, body, chunked)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 668, in request
    self._error_checker(resp, resp_body)
  File "/usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py", line 794, in _error_checker
    raise exceptions.PreconditionFailed(resp_body, resp=resp)
tempest.lib.exceptions.PreconditionFailed: Precondition Failed
Details: Bad URL

Still investigating, meanwhile, adding this test in the skip list

Reviewed: https://review.openstack.org/570533
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=f2eb646202f4d9ef52904cdf840dd7308b450479
Submitter: Zuul
Branch: master

commit f2eb646202f4d9ef52904cdf840dd7308b450479
Author: Arx Cruz <email address hidden>
Date: Fri May 25 10:32:07 2018 +0200

    Adding test_object_services to skip list

    This test is continuously failing on the gates in scenario002

    Change-Id: Ib0d47e7adc72f3a5f9387f87ef11c0ec78f72f06
    Related-Bug: #1773325

Matt Young (halcyondude) on 2018-06-04
tags: removed: quickstart

Reviewed: https://review.openstack.org/580423
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=7ee0e88208a07debd2839a41c56ee9570ed71d40
Submitter: Zuul
Branch: master

commit 7ee0e88208a07debd2839a41c56ee9570ed71d40
Author: Arx Cruz <email address hidden>
Date: Thu Jul 5 12:39:37 2018 -0300

    Removing test from skip list

    The test is no longer failing

    Change-Id: Ib784a47abf5419c188bb17cbe0aacde5a9b4344e
    Related-Bug: 1773325

wes hayutin (weshayutin) on 2018-07-12
Changed in tripleo:
milestone: none → rocky-3
tags: added: alert
removed: promotion-blocker
Ben Nemec (bnemec) wrote :

I take it this was not actually fixed, despite what the commit messages suggest? I know my patches are still failing on it pretty consistently.

wes hayutin (weshayutin) on 2018-07-13
tags: added: promotion-blocker

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/582339
Reason: scenario002 is failing in the gate, this patch needs to be out of the gate until we fix CI. I'll restore this patch in a few hours as the fix is actually in the gate. Please do not restore or recheck.

chandan kumar (chkumar246) wrote :
Download full text (18.1 KiB)

Some more logs from http://logs.openstack.org/18/581918/5/check/tripleo-ci-centos-7-scenario002-multinode-oooq-container/3cf2734/logs/undercloud/var/log/containers/tempest/tempest.log.txt.gz#_2018-07-13_04_47_43_593 giving 412 status code.

2018-07-13 04:47:43.582 81 INFO tempest.lib.common.rest_client [tx0059ce08a31c4dd4bd072-005b482eef ] Request (ObjectTest:setUpClass): 200 GET http://192.168.24.7:8080/info 0.431s
2018-07-13 04:47:43.583 81 DEBUG tempest.lib.common.rest_client [tx0059ce08a31c4dd4bd072-005b482eef ] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '200', u'content-length': '1643', 'content-location': 'http://192.168.24.7:8080/info', u'connection': 'close', u'x-trans-id': 'tx0059ce08a31c4dd4bd072-005b482eef', u'date': 'Fri, 13 Jul 2018 04:47:43 GMT', u'content-type': 'application/json; charset=UTF-8', u'x-openstack-request-id': 'tx0059ce08a31c4dd4bd072-005b482eef'}
        Body: {"formpost": {}, "bulk_delete": {"max_failed_deletes": 1000, "max_deletes_per_request": 10000}, "versioned_writes": {"allowed_flags": ["x-versions-location", "x-history-location"]}, "bulk_upload": {"max_failed_extractions": 1000, "max_containers_per_extraction": 10000}, "container_quotas": {}, "tempurl": {"outgoing_remove_headers": ["x-object-meta-*"], "methods": ["GET", "HEAD", "PUT", "POST", "DELETE"], "incoming_remove_headers": ["x-timestamp"], "allowed_digests": ["sha1", "sha256", "sha512"], "outgoing_allow_headers": ["x-object-meta-public-*"], "incoming_allow_headers": []}, "s3api": {"max_upload_part_num": 1000, "allow_multipart_uploads": true, "max_bucket_listing": 1000, "s3_acl": false, "min_segment_size": 5242880, "max_parts_listing": 1000, "max_multi_delete_objects": 1000}, "ratelimit": {"account_ratelimit": 0.0, "max_sleep_time_seconds": 60.0, "container_ratelimits": [], "container_listing_ratelimits": []}, "slo": {"max_manifest_segments": 1000, "yield_frequency": 10, "min_segment_size": 1, "max_manifest_size": 2097152}, "account_quotas": {}, "staticweb": {}, "swift": {"max_file_size": 5368709122, "account_listing_limit": 10000, "account_autocreate": true, "max_meta_count": 90, "max_meta_value_length": 256, "container_listing_limit": 10000, "extra_header_count": 0, "max_meta_overall_size": 4096, "version": "2.18.1.dev169", "max_meta_name_length": 128, "max_header_size": 8192, "policies": [{"default": true, "name": "Policy-0", "aliases": "Policy-0"}], "max_object_name_length": 1024, "max_account_name_length": 256, "strict_cors_mode": true, "allow_account_management": true, "max_container_name_length": 256}} _log_request_full /usr/lib/python2.7/site-packages/tempest/lib/common/rest_client.py:434
2018-07-13 04:47:43.592 81 INFO tempest.lib.common.rest_client [txb87cc715719a4d748bdbd-005b482eef ] Request (ObjectTest:setUpClass): 412 PUT http://192.168.24.7:8080/tempest-TestContainer-1152572772 0.008s
2018-07-13 04:47:43.592 81 DEBUG tempest.lib.common.rest_client [txb87cc715719a4d748bdbd-005b482eef ] Request - Headers: {'X-Auth-Token': '<omitted>'}
        Body: None
    Response - Headers: {'status': '412', u'content-length': '...

Reviewed: https://review.openstack.org/582339
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=6b05d5ab400bda8377eed3d524a5b354dde79ff5
Submitter: Zuul
Branch: master

commit 6b05d5ab400bda8377eed3d524a5b354dde79ff5
Author: Alex Schultz <email address hidden>
Date: Thu Jul 12 19:00:04 2018 +0000

    Revert "Removing test from skip list"

    This reverts commit 7ee0e88208a07debd2839a41c56ee9570ed71d40.

    It's still failing

    Change-Id: Icd17453092a0c1043b46ff7307f8d48e0e72eb63
    Related-Bug: 1773325

Christian Schwede (cschwede) wrote :

Thanks Chandan, now it is more clear why Swift complains.

2018-07-13 04:47:43.592 81 INFO tempest.lib.common.rest_client [txb87cc715719a4d748bdbd-005b482eef ] Request (ObjectTest:setUpClass): 412 PUT http://192.168.24.7:8080/tempest-TestContainer-1152572772 0.008s

The request uses http://192.168.24.7:8080/tempest-TestContainer-1152572772 as URL, and Swift doesn't know how to handle this.

It should look sth like this: http://192.168.24.7:8080/v1/AUTH_<tenant UUID>/tempest-TestContainer-1152572772

So the "v1/AUTH_<tenant UUID>" part is missing from the Tempest-generated URL. I'm not sure where this URL is generated in Tempest, but I think this is not a Swift issue, but a Tempest one.

Arx Cruz (arxcruz) wrote :

Tempest get these urls from the endpoint publicURL in openstack

wes hayutin (weshayutin) wrote :

@arx @Christian so how to proceed? Arx can you please post a paste of the publicURL printed out by the swift endpoint.

Christian I suppose there are two ways to fix this..
1. hardcode the url path in python-tempestconf
2. fix what is return by endpoint list from keystone.

WDYT?

wes hayutin (weshayutin) wrote :

Is this related at all ?
http://logs.openstack.org/37/579237/1/gate/tripleo-ci-centos-7-undercloud-containers/959e942/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-07-13_20_04_15

Creating a new temporary URL for project service",
2018-07-13 20:04:15 | "stderr: Unable to establish connection to https://192.168.24.2:13808/v1/AUTH_b4a6c3a1aeaf4a90b1fab78bdd523264: ('Connection aborted.', BadStatusLine(\"''\",))",

Fix proposed to branch: master
Review: https://review.openstack.org/583659

Changed in tripleo:
status: Triaged → In Progress

Reviewed: https://review.openstack.org/583659
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=018d057f49bcac509ca67b56b4075a0e44209254
Submitter: Zuul
Branch: master

commit 018d057f49bcac509ca67b56b4075a0e44209254
Author: Arx Cruz <email address hidden>
Date: Wed Jul 18 13:22:53 2018 -0300

    Fix tempest-deployer-input.conf path

    Tempest containerized run as root, and so, although the
    tempest-deployer-input.conf file was being imported to the container,
    the patch was ~/tempest-deployer-input.conf that was pointing to /root
    instead of working_dir, so, tempestconf wasn't getting the defaults from
    the deployer input file, setting the region to RegionOne instead of
    regionOne, and so, the object_store tests were failing, because was
    getting the default endpoint http://.../ instead of http://.../v1/AUTH

    Change-Id: Ic88c9121441b4021da3400933cacc4ff942b727f
    Closes-Bug: 1773325

Changed in tripleo:
status: In Progress → Fix Released
chandan kumar (chkumar246) wrote :
Changed in tripleo:
status: Fix Released → Triaged
assignee: Arx Cruz (arxcruz) → nobody
milestone: rocky-3 → stein-2
tags: removed: alert ci
Arx Cruz (arxcruz) wrote :

Problem seems to be in https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-rocky/56e69c7/logs/undercloud/home/zuul/tempest-setup.sh

You are copying the tempest-deployer-input.conf to $TEMPEST_HOST_DATA that is /var/lib/tempestdata

# Copy all the required files in a temprory directory
export TEMPEST_HOST_DATA='/var/lib/tempestdata'

if [ ! -d $TEMPEST_HOST_DATA ]
then
    sudo mkdir -p $TEMPEST_HOST_DATA
fi

sudo cp $RCFILE \
        /home/zuul/skip_file \
            /home/zuul/whitelist_file.conf \
            /home/zuul/tempest-deployer-input.conf \
        /home/zuul/tempest_container.sh \
    $TEMPEST_HOST_DATA

And executing the tempestconf tool looking for /var/lib/tempest/tempest:

$TEMPESTCONF --out etc/tempest.conf \
    --deployer-input /var/lib/tempest/tempest/tempest-deployer-input.conf \
  --network-id $public_net_id \

Also the order is switched, you should set the variables, and then run tempestconf, here you're copying the deployer input file to $TEMPEST_HOST_DATA AFTER run tempestconf, which makes no sense for me

Fix proposed to branch: master
Review: https://review.openstack.org/618994

Changed in tripleo:
assignee: nobody → chandan kumar (chkumar246)
status: Triaged → In Progress

Change abandoned by Juan Antonio Osorio Robles (<email address hidden>) on branch: master
Review: https://review.openstack.org/618994
Reason: clearing the gate

Reviewed: https://review.openstack.org/618994
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=d9a66b56b4898e2707e1e358a1eba3dddddbe516
Submitter: Zuul
Branch: master

commit d9a66b56b4898e2707e1e358a1eba3dddddbe516
Author: Chandan Kumar <email address hidden>
Date: Tue Nov 20 17:31:31 2018 +0530

    Fixed path for tripleo deployer input file

    * While porting the validate-tempest to podman[1], we added tempest_data
      variable to keep tempest related input data files while running in
      container, All the dir path in the script got updated but tripleo
      deployer input path was left to tempest_dir but we were keeping
      the files in tempest_data dir due to which tripleo deployer
      input file was not passed leading to the failure of object
      storage tests.

    [1.] https://review.openstack.org/605356

    Change-Id: I132009d3f00bb4bcff3b21def0edd568bfab14ec
    Closes-Bug:#1773325

Changed in tripleo:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers