Comment 0 for bug 1577893

MOS 9.0 systest run 661 on iso 201 (https://packaging-ci.infra.mirantis.net/job/9.0-pkg-systest-ubuntu/661/console) failed with the following traceback:

FAIL: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/tests/test_ceph.py", line 502, in ceph_rados_gw
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 455, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 440, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 491, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 498, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 382, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 817, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval, timeout=timeout)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 319, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[database/3], Task[database/2] Stopping the deployment process!'

In the remote/puppet-apply.log logs on node 3 there are a couple of failures:

2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) Fetched 8564 kB in 1s (8512 kB/s)
2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) dpkg: error: dpkg status database is locked by another process
2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) E: Sub-process /usr/bin/dpkg returned an error code (2)
2016-04-29T06:26:39.718277+00:00 err: (Package[pacemaker](provider=apt_fuel)) Fetched 8564 kB in 1s (8512 kB/s)
2016-04-29T06:26:39.718409+00:00 err: (Package[pacemaker](provider=apt_fuel)) dpkg: error: dpkg status database is locked by another process
2016-04-29T06:26:39.718409+00:00 err: (Package[pacemaker](provider=apt_fuel)) E: Sub-process /usr/bin/dpkg returned an error code (2)

and

2016-04-29T06:36:44.766309+00:00 err: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

On node 2 there is a following error:

2016-04-29T06:40:57.718177+00:00 notice: (/Stage[main]/Cluster::Mysql/Exec[wait-initial-sync]/returns) ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
2016-04-29T06:40:57.721064+00:00 err: (/Stage[main]/Cluster::Mysql/Exec[wait-initial-sync]) Failed to call refresh: mysql -uclustercheck -pnOTvDD8mPReXplHCuDPcUI2X -Nbe "show status like 'wsrep_local_state_comment'" | grep -q -e Synced && sleep 10 returned 1 instead of one of [0]

There are also lots of "32 (broken pipe)" errors in mysqld logs.