Fuel for OpenStack

database tasks failed during deployment

Bug #1577893 reported by Vladyslav Drok on 2016-05-03

This bug report is a duplicate of: Bug #1571763: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) errors during cluster deployment.. Edit Remove

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	Fuel for OpenStack	New	Undecided	Unassigned

Bug Description

MOS 9.0 systest run 661 on iso 201 (https://packaging-ci.infra.mirantis.net/job/9.0-pkg-systest-ubuntu/661/console) failed with the following traceback:

FAIL: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 120, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/tests/test_ceph.py", line 502, in ceph_rados_gw
    self.fuel_web.deploy_cluster_wait(cluster_id)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 455, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 440, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 491, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 498, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 382, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 817, in deploy_cluster_wait
    self.assert_task_success(task, interval=interval, timeout=timeout)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 319, in assert_task_success
    task["name"], task['status'], 'ready', _message(task)
AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[database/3], Task[database/2] Stopping the deployment process!'

In the remote/puppet-apply.log logs on node 3 there are a couple of failures:

2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) Fetched 8564 kB in 1s (8512 kB/s)
2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) dpkg: error: dpkg status database is locked by another process
2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) E: Sub-process /usr/bin/dpkg returned an error code (2)
2016-04-29T06:26:39.718277+00:00 err: (Package[pacemaker](provider=apt_fuel)) Fetched 8564 kB in 1s (8512 kB/s)
2016-04-29T06:26:39.718409+00:00 err: (Package[pacemaker](provider=apt_fuel)) dpkg: error: dpkg status database is locked by another process
2016-04-29T06:26:39.718409+00:00 err: (Package[pacemaker](provider=apt_fuel)) E: Sub-process /usr/bin/dpkg returned an error code (2)

and

2016-04-29T06:36:44.766309+00:00 err: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

On node 2 there is a following error:

2016-04-29T06:40:57.718177+00:00 notice: (/Stage[main]/Cluster::Mysql/Exec[wait-initial-sync]/returns) ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)
2016-04-29T06:40:57.721064+00:00 err: (/Stage[main]/Cluster::Mysql/Exec[wait-initial-sync]) Failed to call refresh: mysql -uclustercheck -pnOTvDD8mPReXplHCuDPcUI2X -Nbe "show status like 'wsrep_local_state_comment'" | grep -q -e Synced && sleep 10 returned 1 instead of one of [0]

There are also lots of "32 (broken pipe)" errors in mysqld logs and these:

2016-04-29T14:25:07.973435+00:00 err: 2016-04-29 14:25:08 32501 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.109.21.4:4444/xtrabackup_sst' --auth 'root:x8LrXWoKZMUfBtuQy3PnJ7qB' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '6864c121-0e13-11e6-b5b6-a3c80f4596ac:0'
2016-04-29T14:25:07.973936+00:00 err: 2016-04-29 14:25:08 32501 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.109.21.4:4444/xtrabackup_sst' --auth 'root:x8LrXWoKZMUfBtuQy3PnJ7qB' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '6864c121-0e13-11e6-b5b6-a3c80f4596ac:0': 22 (Invalid argument)
2016-04-29T14:25:07.973936+00:00 err: 2016-04-29 14:25:08 32501 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.109.21.4:4444/xtrabackup_sst' --auth 'root:x8LrXWoKZMUfBtuQy3PnJ7qB' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '6864c121-0e13-11e6-b5b6-a3c80f4596ac:0'