[9.x][BVT] Multiple authorization failures

[Vitaly Sedelnik]

BVT job [1] failed with the following errors:

ssertionError: Failed 17 OSTF tests; should fail 0 tests. Names of failed tests:
  - Check if amount of tables in databases is the same on each node (failure) Can list tables Please refer to OpenStack logs for more details.
  - Check galera environment state (failure) Actual value - 3,
  - Check pacemaker status (failure) On the controller node-4.test.domain.local, resource clone_p_mysqld is active but failed to start (managed).. Please refer to OpenStack logs for more details.
  - Request stack list (error)
  - Request active services list (failure) Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
  - Request user list (failure) Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
  - Check that required services are running (failure) Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
  - Request list of networks (failure) Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
  - Check create, update and delete image actions using Glance v2 (failure) Error creating image. Please refer to Openstack logs for more information. Please refer to OpenStack logs for more details.
  - Create volume and boot instance from it (failure) Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
  - Create volume and attach it to instance (failure) Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
  - Check network connectivity from instance via floating IP (error)
  - Create keypair (failure) Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
  - Create security group (failure) Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
  - Check network parameters (failure) Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
  - Launch instance (failure) Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.
  - Launch instance with file injection (failure) Authorization failure. Please provide the valid credentials for your OpenStack environment, and reattempt.

[1] https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/690/console

[Vitaly Sedelnik (vsedelnik)]

Diagnostic snapshot is https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/690/artifact/logs/fail_error_ceph_rados_gw-fuel-snapshot-2016-12-27_13-54-50.tar

[Boris Bobrov (bbobrov)]

There are many errors like this:

2016-12-27 13:52:58.993 14176 ERROR keystone.common.wsgi DBConnectionError: (_mysql_exceptions.OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 11")

Looks like something is wrong with MySQL. MOS Linux, could you take a look please?

[Oleksiy Molchanov (omolchanov)]

Mysql was shut down.

http://paste.openstack.org/show/593472/
http://paste.openstack.org/show/593473/