[10.0] [Tempest] Ceph storage tests ara failed on SSL env

[Yury Tregubov]

Since about mos10 build #1530 Tempest tests are failed to setup container:

Traceback (most recent call last):
  File "tempest/test.py", line 270, in setUpClass
    six.reraise(etype, value, trace)
  File "tempest/test.py", line 263, in setUpClass
    cls.resource_setup()
  File "tempest/api/object_storage/test_object_services.py", line 38, in resource_setup
    cls.container_name = cls.create_container()
  File "tempest/api/object_storage/base.py", line 80, in create_container
    cls.container_client.create_container(container_name)
  File "tempest/services/object_storage/container_client.py", line 46, in create_container
    resp, body = self.put(url, body=None, headers=headers)
  File "tempest/lib/common/rest_client.py", line 340, in put
    return self.request('PUT', url, extra_headers, headers, body, chunked)
  File "tempest/lib/common/rest_client.py", line 664, in request
    self._error_checker(resp, resp_body)
  File "tempest/lib/common/rest_client.py", line 746, in _error_checker
    resp=resp)
tempest.lib.exceptions.UnexpectedContentType: Unexpected content type provided
Details: 404

Diagnostic snapshot is https://drive.google.com/a/mirantis.com/file/d/0B1Crk-sAvGanNUc5aVVxZ2FFbXc/view?usp=sharing

Tempest logs are attached

[Oleksiy Molchanov (omolchanov)]

Radoslaw, can you check this?

[Alexey Shtokolov (ashtokolov)]

Set to High. If it blocks tempest again, please raise to Critical.
Waiting for Storage team feedback.

[Radoslaw Zarzynski (rzarzynski)]

In the provided Tempest log we can find the pair of HTTP
request and response that caused the test failure [1].

Tempest has issued PUT to /swift/v1/TestContainer-800644525.
However, RadosGW clearly dispatched the request to its code
path for the S3 or admin APIs. The evidence is the response
containing:
 * the AWS-specific "x-amz-request-id" header in place of
   Swift's "X-Trans-Id",
 * the body with XML-formatted, S3-specific error message.

Each of these items can be generated only in the S3-related
paths in RadosGW [2].

Although in the provided snapshot all RadosGW's logs are empty
for all controllers, fortunately we still can extract the bucket
name from the error message:

<BucketName>public.fuel.local</BucketName>

In the Apache's access log for RadosGW we can observe:

  10.109.4.6 - - [29/Mar/2017:00:09:22 +0000] "PUT /swift/v1/TestContainer-800644525 HTTP/1.1" 404 227 "-" "-"

This suggests we don't have a misbehaving HTTP client here.
Possibly the problem is in the FastCGI interconnection between
Apache and RadosGW. I verified that the configurables at RGW's
side look OK.

Have we updated Apache (including modules) or changed its
configuration or updated it (of its modules) recently?

[1] https://gist.github.com/rzarzynski/de2cd2b0bea12d9af39829381dcb2840
[2] https://github.com/ceph/ceph/blob/hammer/src/rgw/rgw_rest.cc#L503-L511

[Oleksiy Molchanov (omolchanov)]

Radoslaw, yesterday we moved from apache to civitweb, let's check results for the tempest for such installation.

[Oleksiy Molchanov (omolchanov)]

I got confirmation that it was not reproduced with civitweb.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/451786

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Polenchuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/451786
Reason: rgw_dns_name option has no effect with civetweb enabled.