[Ceph] [tempest] All tests for ObjectStorage API failed for clouds with Ceph RadosGW

[Timur Nurlygayanov]

We can see that 130 Tempest tests failed for OpenStack clouds, which were deployed by MOS and looks like the reason of these fails in non-full support of SwiftAPI in Ceph RadosGW.

Steps To Reproduce:
1. Deploy cluster with Ceph RadosGW
2. Run full Tempest test suite using "rally verify"

Expected Result:
All Tempest tests passed without any issues.

Actual Result:
Many Tempest tests failed because RadosGW API doesn't support the same return codes as Swift API, in the result Tempest tests failed.

This issue reproduced on 6.0, 6.1, 7.0, 8.0 and 9.0 versions.

Note:
Ceph team work on this, we can see the information about the progress here:
https://docs.google.com/a/mirantis.com/spreadsheets/d/1LZx3ZRCK-ezaKddyE6HgU-Xau2DOZKtAYLrtkit0jPU/edit#gid=0

[Timur Nurlygayanov (tnurlygayanov)]

Why is it not fixed?

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Sergii Turivnyi (sturivnyi)]

This bug also affects heat. We have 2 failed tests:
test_metadata in tempest.api.orchestration.stacks.test_swift_resources.SwiftResourcesTestJSON
test_acl in tempest.api.orchestration.stacks.test_swift_resources.SwiftResourcesTestJSON

[Timur Nurlygayanov (tnurlygayanov)]

We have agreed to add the release note about the issue and skip the tests because RadosGW and Swift API has different APIs

[Maksym Shalamov (mshalamov)]

Hello all,

This issue is associated with both tests behavior and product functionality. We are using the same test created for Swift for testing radosGW

However Swift, besides implementing the OpenStack Object Storage API v1, provides a bunch extensions like SLO/DLO (static/dynamic large objects), bulk operations (inc. server-side archive extraction), staticweb and many more.

Swift internally utilizes a pipeline-based architecture. Component delivering the base of OSOS API (the "proxy-server" WSGI application) could be seen as the last stage of such pipeline. Between it and user a lot of additional middleware modules exist and all of it are interconnected with the WSGI interface.

Based on all described above we able to say that:
for today we have a serious difference between Swift and Rados Gateway functionality.
reimplementing described above features for Rados Gateway could be “painful”.
maintaining this features in case if implemented would be another challenge.

From another side, even if we implement it, we will have another “big restrictions” - our configuration will be frozen and un-upgradeable, any additional third party extensions will not be available for Rados Gateway users.

Based on all of the above-mentioned, we are able to аllocate three main reasons for the falling tests:
issues in implementation of OpenStack Object Storage API v1 for Rados Gateway;
no similar extensions for Rados Gateway that exist for Swift;
in tempest tests for object storage present verification of functionality that did not described in official documentation for Swift API.

RefStack [1] can validate OpenStack Interoperability [2]. For Object store there is a short subset of basic Tempest tests that should be green.

Object Storage Capabilities objectstore-object-get✓ ✓objectstore-object-delete✓ ✓objectstore-object-copy✓ ✓objectstore-object-access✓ ✓objectstore-object-create✓ ✓objectstore-object-put✓ ✓objectstore-object-upload✓ ✓objectstore-object-versioned✓ ✓objectstore-temp-url-get✓ ✓

Based on all of the above-mentioned, we have agreed to add the release note about the issue and skip the unrequired tests

For more details related to executions of the tempest tests against Rados GW, please take a look into [3]

[1] https://wiki.openstack.org/wiki/RefStack
[2] http://www.openstack.org/brand/interop/
[3] http://cz7776.bud.mirantis.net:8080/jenkins/view/TEMPEST-9.0/

[Timur Nurlygayanov (tnurlygayanov)]

please do not assign such defects directly to OpenStack projects. It should be in MOS.

[Timur Nurlygayanov (tnurlygayanov)]

I've described a separate bug for Tempest project:
https://bugs.launchpad.net/tempest/+bug/1590505

Let's track release notes in this bug and proper fix in bug inside Tempest project.

[Michele Fagan (michelefagan)]

@Maksym Shalamov: Are you able to provide the text for the release note re. Swift and Rados Gateway functionality? Thanks.

[Maksym Shalamov (mshalamov)]

@Michele Fagan: The text will be created by the end of week.

[Maksym Shalamov (mshalamov)]

Appropriate document created

https://docs.google.com/document/d/1lGK8isU_66W0Hdl7KPlKcGG-yqfzIonhRoCT2TRKP9Q/edit#

[Evgeny Konstantinov (evkonstantinov)]

it's in the KB