[regression] 1-Swift related Tempest tests failed on MOS 9.0 Mitaka builds

[Timur Nurlygayanov]

Detailed bug description:
[regression] 1-Swift related Tempest tests failed on MOS 9.0 Mitaka #64.

Steps to reproduce:
1. Deploy OpenStack cluster with the following configuration: HA, Neutron VLANs, Cinder LVM, Swift (without Ceph).
2. Run full Tempest test suite using Rally tool (rally verify)

Expected results:
All tests passed

Actual result:
1-Swift related Tempest tests failed

Reproducibility:
100%

Workaround:
We don't know the workaround yet.

Impact:
Some Swift functionality doesn't work, all swift users are affected.

Description of the environment:
- Operation system: Ubuntu 14.04
- Versions of components: MOS 9.0 Mitaka #64
- Reference architecture: HA, Neutron VLANs, Cinder LVM, Swift (without Ceph)
- Network model: Neutron VLANs
- Related projects installed: -

Additional information:
You can find the test report by the following link:
https://mirantis.testrail.com/index.php?/runs/view/7052&group_by=cases:section_id&group_order=asc&group_id=16871

[Oleksiy Butenko (obutenko)]

http://paste.openstack.org/show/489997/

[Maksym Shalamov (mshalamov)]

Hello all,

According to results of my investigation I supposed that root cause of the test failing related with issues in Tempest configuration and in more detail: `CONF.object_storage_feature_enabled.container_sync.

 Issue [1] is duplicate of this issue, because both issues has similar root cause.

[1] https://bugs.launchpad.net/mos/+bug/1558689

[Maksym Shalamov (mshalamov)]

I keep working on the task.
Due date: 2016-05-06

[Maksym Shalamov (mshalamov)]

Hello all,

During the investigation of the issue we found the root cause of the issue. We suppose that issue related with problems in Swift configuration.

Based on [1] we are able to say that in configuration files missing following:

1. proxy-server.conf (/etc/swift/proxy-server.conf)
The varible pipeline from config section [pipeline:main] should contain container_sync, but the varible pipeline from config section [pipeline:main] is not contain container_sync parameter.

[pipeline:main]
pipeline = catch_errors crossdomain healthcheck cache bulk tempurl ratelimit formpost swift3 s3token authtoken keystone staticweb container_quotas account_quotas slo proxy-server

2. container-sync-realms.conf (/etc/swift/container-sync-realms.conf)
The config file should contain specifying the allowable cluster/clusters and their information, but config file contain only commented text

Please @mos-swift confirm that results of our investigation is correct and perform appropriate action to resolve this issue.

[Maksym Shalamov (mshalamov)]

[1]http://docs.openstack.org/developer/swift/overview_container_sync.html

[Maksym Shalamov (mshalamov)]

Verified on:

cat /etc/fuel_build_id:
 278
cat /etc/fuel_build_number:
 278
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6343.noarch
 fuel-misc-9.0.0-1.mos8332.noarch
 fuel-mirror-9.0.0-1.mos133.noarch
 shotgun-9.0.0-1.mos88.noarch
 fuel-openstack-metadata-9.0.0-1.mos8677.noarch
 fuel-notify-9.0.0-1.mos8332.noarch
 fuel-ostf-9.0.0-1.mos930.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8677.noarch
 python-fuelclient-9.0.0-1.mos313.noarch
 fuel-9.0.0-1.mos6343.noarch
 fuel-utils-9.0.0-1.mos8332.noarch
 fuel-nailgun-9.0.0-1.mos8677.noarch
 rubygem-astute-9.0.0-1.mos742.noarch
 fuel-library9.0-9.0.0-1.mos8332.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-agent-9.0.0-1.mos274.noarch
 fuel-ui-9.0.0-1.mos2676.noarch
 fuel-setup-9.0.0-1.mos6343.noarch
 nailgun-mcagents-9.0.0-1.mos742.noarch
 python-packetary-9.0.0-1.mos133.noarch
 fuelmenu-9.0.0-1.mos269.noarch
 fuel-bootstrap-cli-9.0.0-1.mos274.noarch
 fuel-migrate-9.0.0-1.mos8332.noarch

[Timur Nurlygayanov (tnurlygayanov)]

Alexey, are you going to fix the issue or it is better to assign the issue to another engineer?

[Denis Egorenko (degorenko)]

Related fix for puppet-swift: https://review.openstack.org/#/c/319949/

[Denis Egorenko (degorenko)]

Fix: https://review.openstack.org/322814

[Timur Nurlygayanov (tnurlygayanov)]

it is in progress for master branch only. We need to prepare the backport for stable/mitaka as well.

[Ivan Berezovskiy (iberezovskiy)]

Timur, actually it's in progress for 9.0 as well. Upstream patch is here https://review.openstack.org/#/c/323409/

[Denis Egorenko (degorenko)]

Mitaka: https://review.openstack.org/#/c/325134/

[Timur Nurlygayanov (tnurlygayanov)]

Ok, the issue was fixed, marked as FixReleased.
Please reopen the issue if it wasn't completely fixed or if it will be reproduced again.

[Vladimir Jigulin (vjigulin)]

Probably same issue: #1591175