[regression] 7 Glance-related Tempest tests failed on MOS 9.0 Mitaka builds

[Timur Nurlygayanov]

Detailed bug description:
7 Glance-related Tempest tests failed on MOS 9.0 Mitaka builds.

The list of failed tests:
test_get_image_without_membership
test_index_container_format
test_index_min_size
test_index_max_size
test_index_disk_format
test_index_name
test_index_no_params

Steps to reproduce:
1. Deploy environment with MOS 9.0 Mitaka #64.
2. Run full Tempest test suite.

Expected results:
All tests passed

Actual result:
7 Glance-related Tempest tests failed

Reproducibility:
100 %, reproduced on 3 different configurations.

Workaround:
We don't know the workaround.

Impact:
 All MOS users are affected.

Description of the environment:
- Operation system: Ubuntu 14.04
- Versions of components: -
- Reference architecture: Neutron VLAN, HA, Cinder LVM, Swift
- Network model: Neutron VLAN
- Related projects installed: -
Additional information:
you can find the test report by the following link:
https://mirantis.testrail.com/index.php?/runs/view/7052&group_by=cases:section_id&group_order=asc&group_id=16864

[Timur Nurlygayanov (tnurlygayanov)]

This issue didn't reproduce on MOS 7.0 and 8.0, it is actual only for MOS 9.0.

So, it is regression issue.

[Olga Klochkova (oklochkova)]

Bug is reproduced from Horizon. This is logs after add Image:

2016-03-09 12:13:46.548 26064 ERROR swiftclient [req-e4382f25-8662-42f4-a9f8-fc7e6054d15b 1d7f772a33664e3c8c4d6955f2daf096 f36c9e002cd546dd87dd407bfc9bea13 - - -] Container HEAD failed: http://10.109.7.10:8080/v1/AUTH_f36c9e002cd546dd87dd407bfc9bea13/glance 404 Not Found
2016-03-09 12:13:46.548 26064 ERROR swiftclient Traceback (most recent call last):
2016-03-09 12:13:46.548 26064 ERROR swiftclient File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1491, in _retry
2016-03-09 12:13:46.548 26064 ERROR swiftclient service_token=self.service_token, **kwargs)
2016-03-09 12:13:46.548 26064 ERROR swiftclient File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 855, in head_container
2016-03-09 12:13:46.548 26064 ERROR swiftclient http_response_content=body)
2016-03-09 12:13:46.548 26064 ERROR swiftclient ClientException: Container HEAD failed: http://10.109.7.10:8080/v1/AUTH_f36c9e002cd546dd87dd407bfc9bea13/glance 404 Not Found
2016-03-09 12:13:46.548 26064 ERROR swiftclient
2016-03-09 12:13:46.553 26064 INFO glance_store._drivers.swift.store [req-e4382f25-8662-42f4-a9f8-fc7e6054d15b 1d7f772a33664e3c8c4d6955f2daf096 f36c9e002cd546dd87dd407bfc9bea13 - - -] Creating swift container glance
2016-03-09 12:13:47.927 26064 INFO glance.api.v1.images [req-e4382f25-8662-42f4-a9f8-fc7e6054d15b 1d7f772a33664e3c8c4d6955f2daf096 f36c9e002cd546dd87dd407bfc9bea13 - - -] Uploaded data of image df0fb147-6865-4494-9a18-965ee4862daf from request payload successfully.

[Kairat Kushaev (kkushaev)]

These logs are ok, when glance cannot find a container in swift (=no found error) we are creating that container and upload data into that container (Uploaded data of image df0fb147-6865-4494-9a18-965ee4862daf from request payload successfully.). So link to logs would be more appropriate here.

[Kairat Kushaev (kkushaev)]

Tried to reproduce on devstack/master with no success.
It is either 9.0 specific issue(can't reproduce on current M master) or tempest specific issue.
Waiting for 9.0 instance to proceed with bug analysis.

[Kairat Kushaev (kkushaev)]

Looks like the problem is related to glance-registry-paste.ini.
Here is 9.0 configuration:
https://paste.mirantis.net/show/1975/
You can see the default flavor was generated twice:
# Use this pipeline for keystone auth
[pipeline:glance-registry-keystone]
pipeline = healthcheck osprofiler authtoken context registryapp

[pipeline:glance-registry-keystone]
pipeline = osprofiler unauthenticated-context registryapp

Apparently, the last pipeline is considered as actual. It leads to situation where unauthenticated context is used as default.
unauthenticated means that we can access all images in registry.
So we need to delete the last condition from glance-registry-paste.ini.

[Kairat Kushaev (kkushaev)]

please note that the same situation is here:
[pipeline:glance-registry-caching]
pipeline = osprofiler unauthenticated-context registryapp

[pipeline:glance-registry-keystone+caching]
pipeline = osprofiler unauthenticated-context registryapp

[pipeline:glance-registry-cachemanagement]
pipeline = osprofiler unauthenticated-context registryapp

[pipeline:glance-registry-keystone+cachemanagement]
pipeline = osprofiler unauthenticated-context registryapp

[Denis Egorenko (degorenko)]

We have outdated patch in glance package: fix-default-glance-registry-paste.ini.patch. We need to remove it.

[Kairat Kushaev (kkushaev)]

Raising to Critical cause it lead to master bvt fails according to :
https://bugs.launchpad.net/fuel/+bug/1556068
It also looks like this bug is also related:
https://bugs.launchpad.net/mos/+bug/1557494

[Sergii Turivnyi (sturivnyi)]

Can't reproduce on CI

[Timur Nurlygayanov (tnurlygayanov)]

The bug was actually fixed.

Ivan, could you please share the lin to the fix?