Failing cinder, nova, heat, glance, neutron, keystone db tasks

[Vladyslav Drok]

Master pkg systest https://packaging-ci.infra.mirantis.net/job/master-pkg-systest-ubuntu/768/console failed with the following traceback:

FAIL: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/tests/test_ceph.py", line 502, in ceph_rados_gw
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 455, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 440, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 491, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 498, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 382, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 827, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval, timeout=timeout)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/master-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 321, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[cinder-db/1], Task[heat-db/1], Task[nova-db/1], Task[glance-db/1], Task[neutron-db/1], Task[keystone-db/1] Stopping the deployment process!'

In the node-1/var/log/puppet.log there are following entries for each service task that failed:
2016-05-06 05:53:13 +0000 /Stage[main]/Keystone::Db::Mysql/Openstacklib::Db::Mysql[keystone]/Mysql_database[keystone]/ensure (err): change from absent to present failed: Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe create database if not exists `keystone` character set `utf8` collate `utf8_general_ci`' returned 1: ERROR 1213 (40001) at line 1: Deadlock found when trying to get lock; try restarting transaction

[Vladyslav Drok (vdrok)]

More failures:

https://packaging-ci.infra.mirantis.net/job/master-pkg-systest-ubuntu/741/
https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.smoke_neutron/159/
https://product-ci.infra.mirantis.net/view/10.0/job/10.0.main.ubuntu.smoke_neutron/158/

[Bogdan Dobrelya (bogdando)]

The errors like
"2016-05-06T05:55:19.208238+00:00 err: 2016-05-06 05:55:19 12448 [Warning] WSREP: Send action {(nil), 261, TORDERED} returned -103 (Software caused connection abort)
2016-05-06T05:55:19.208238+00:00 err: 2016-05-06 05:55:19 12448 [Warning] WSREP: TO isolation failed for: 3, sql: create database if not exists `heat` character set `utf8` collate `utf8_general_ci`. Check wsrep connection state and retry the query."

looks like a MySQL/galera specific issue, I'm not sure how to debug it. May be Sergey G. may shed some light.

[Bogdan Dobrelya (bogdando)]

This ISO is old, there was a non relevant code produced the error messages like:
2016-05-06T05:36:36.654679+00:00 warning: WARNING: p_mysqld: mysql_monitor(): found and purged a stale sst_in_progress file

There is no more such a logic in a master branch. This is a duplicate.