[10.0-mitaka]bvt_2: Failed 6 OSTF tests

[Alexey Golubev]

Jenkins job: https://product-ci.infra.mirantis.net/job/10.0-mitaka.main.ubuntu.bvt_2/56

From sys_test.log:

2016-07-06 00:03:58,338 - INFO decorators.py:318 -- <<<<<****************************************************************************************************>>>>>
2016-07-06 00:03:58,338 - INFO decorators.py:319 -- Failed in method error_ceph_rados_gw. Make snapshot: error_ceph_rados_gw
2016-07-06 00:03:58,338 - INFO decorators.py:329 -- You could revert and ssh to master node: [source /home/jenkins/venv-nailgun-tests-2.9/bin/activate; dos.py revert-resume 10.0-mitaka.main.ubuntu.bvt_2.56 error_ceph_rados_gw && ssh root@10.109.5.2]
2016-07-06 00:03:58,338 - INFO decorators.py:331 -- <<<<<****************************************************************************************************>>>>>
...
AssertionError: Failed 6 OSTF tests; should fail 0 tests. Names of failed tests:
  - Create volume and boot instance from it (failure) TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant.
  - Create volume and attach it to instance (failure) TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant.
  - Check network connectivity from instance via floating IP (failure) TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant.
  - Launch instance (failure) TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant.
  - Launch instance with file injection (failure) TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant.
  - Launch instance, create snapshot, launch instance from snapshot (failure) TestVM image not found. Please, download http://download.cirros-cloud.net/0.3.1/cirros-0.3.1-x86_64-disk.img image and register it in Glance with name 'TestVM' as 'admin' tenant.

[Tatyanka (tatyana-leontovich)]

looks like related to glance-registry:
http://paste.openstack.org/show/526557/

So as result when ostf request image list from glance it comes to it empty

[Igor Shishkin (teran)]

It's not clear where the issue come from and how it's related to Infra(or specifically DevOps) team.
Please triage according to issue cause since based on logs and comments it's not related to Infra team.

[Roman Prykhodchenko (romcheg)]

After briefly looking at glance-registry.log files I have found that the real reason may be that keystone was sending 500 when Glance Registry service tried to verify tokens.

[Roman Prykhodchenko (romcheg)]

This bug is probably already fixed. Could someone from mos-glance please check it out?

[Kairat Kushaev (kkushaev)]

Hi Roman,
I am not sure that the root cause is in Glance. I have investigated the problem a bit. It looks like in some cases Keystone auth was failing with glance-registry. I digged into keystone logs and found the following for g-registry requests:
https://paste.mirantis.net/show/2561/
I found this in keystone-admin.log on node-3 and node-5.
IIUC, there is something wrong with db configuration for such kind of error, please correct me if I am wrong.

[Alexander Makarov (amakarov)]

From the logs above it's obvious that the loss of mysql connection is the reason of problem.

[Dmitry Pyzhov (dpyzhov)]

This is a duplicate of one of mysql bugs that are already fixed.

[Alexander Ignatov (aignatov)]

An issues had been reproduced again with the same syphtoms in

https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/929/console

[Oleksiy Molchanov (omolchanov)]

Marking as Invalid, job mentioned in comment #9 is already unavailable.