OSTF tests fail because of keystone DB errors

[Matthew Mosesohn]

Custom CentOS BVT 201 failed on creating instance flavor because of this error:
AuthorizationFailure: Authorization Failed: An unexpected error prevented the server from fulfilling your request: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user \nWHERE user.name = %s AND user.domain_id = %s' ('admin', 'default') (Disable debug mode to suppress these details.) (HTTP 500)

This is not the first time and it's an intermittent bug for sure. I believe we can work around this by waiting for galera ready state. This could be done either within the framework of OSTF (run Galera check with a backoff timer before any OpenStack tests) or within fuel system tests itself to run a similar check before starting OSTF.

[Tatyanka (tatyana-leontovich)]

I disagree with such behavior. It is really un-normal situation when fuel says that cluster is ready but galera is not synced. If we want to get iso only - we can not run ostf for some time

[Matthew Mosesohn (raytrac3r)]

It looks like we're seeing this: https://review.openstack.org/#/c/125079/
Oslo changes for Juno/Kilo resulted in some poor reconnect behavior with MySQL. This patch should fix it.

[OSCI Robot (oscirobot)]

Package oslo.db has been built from changeset: http://gerrit.mirantis.com/30006
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-30006/centos

[OSCI Robot (oscirobot)]

Package oslo.db has been built from changeset: http://gerrit.mirantis.com/30006
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-30006/ubuntu

[OSCI Robot (oscirobot)]

Package oslo.db has been built from changeset: http://gerrit.mirantis.com/30006
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable/ubuntu

[OSCI Robot (oscirobot)]

Package oslo.db has been built from changeset: http://gerrit.mirantis.com/30006
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable/centos

[Tatyanka (tatyana-leontovich)]

verified {"build_id": "2015-02-12_10-39-11", "ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f", "build_number": "115", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-12_10-39-11", "ostf_sha": "f9c37d0876141e1550eb4e703a8e500cd463282f", "build_number": "115", "api": "1.0", "nailgun_sha": "78e1fa50e38efc1001ddfe0565a55e9e176ff5f6", "production": "docker", "python-fuelclient_sha": "61431ed16fc00039a269424bdbaa410277eff609", "astute_sha": "2159855ba7b82956ac0787a4e7be053105c4c1f1", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "892c80aa9adc9f53e9b3061d4754203953a84db7", "fuellib_sha": "592df3ba1dbfba6c6c84f90ff36b0c4c697934d3"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "78e1fa50e38efc1001ddfe0565a55e9e176ff5f6", "production": "docker", "python-fuelclient_sha": "61431ed16fc00039a269424bdbaa410277eff609", "astute_sha": "2159855ba7b82956ac0787a4e7be053105c4c1f1", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "892c80aa9adc9f53e9b3061d4754203953a84db7", "fuellib_sha": "592df3ba1dbfba6c6c84f90ff36b0c4c697934d3"}

[Tatyanka (tatyana-leontovich)]

http://jenkins-product.srt.mirantis.net:8080/view/6.0.1/ all bvt also are passed