Activity log for bug #1577893

Date Who What changed Old value New value Message
2016-05-03 18:38:37 Vladyslav Drok bug added bug
2016-05-03 18:41:25 Vladyslav Drok attachment added Diagnostic snapshot https://bugs.launchpad.net/fuel/+bug/1577893/+attachment/4654736/+files/fail_error_ceph_rados_gw-fuel-snapshot-2016-04-29_07-06-54.tar.xz
2016-05-03 18:51:23 Vladyslav Drok description MOS 9.0 systest run 661 on iso 201 (https://packaging-ci.infra.mirantis.net/job/9.0-pkg-systest-ubuntu/661/console) failed with the following traceback: FAIL: Deploy ceph HA with RadosGW for objects ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func compatability.capture_type_error(s_func) File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error func() File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func func(test_case.state.get_state()) File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 120, in wrapper result = func(*args, **kwargs) File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/tests/test_ceph.py", line 502, in ceph_rados_gw self.fuel_web.deploy_cluster_wait(cluster_id) File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 455, in wrapper result = func(*args, **kwargs) File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 440, in wrapper result = func(*args, **kwargs) File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 491, in wrapper return func(*args, **kwargs) File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 498, in wrapper result = func(*args, **kwargs) File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 382, in wrapper return func(*args, **kwargs) File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 817, in deploy_cluster_wait self.assert_task_success(task, interval=interval, timeout=timeout) File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped result = func(*args, **kwargs) File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 319, in assert_task_success task["name"], task['status'], 'ready', _message(task) AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[database/3], Task[database/2] Stopping the deployment process!' In the remote/puppet-apply.log logs on node 3 there are a couple of failures: 2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) Fetched 8564 kB in 1s (8512 kB/s) 2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) dpkg: error: dpkg status database is locked by another process 2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) E: Sub-process /usr/bin/dpkg returned an error code (2) 2016-04-29T06:26:39.718277+00:00 err: (Package[pacemaker](provider=apt_fuel)) Fetched 8564 kB in 1s (8512 kB/s) 2016-04-29T06:26:39.718409+00:00 err: (Package[pacemaker](provider=apt_fuel)) dpkg: error: dpkg status database is locked by another process 2016-04-29T06:26:39.718409+00:00 err: (Package[pacemaker](provider=apt_fuel)) E: Sub-process /usr/bin/dpkg returned an error code (2) and 2016-04-29T06:36:44.766309+00:00 err: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) On node 2 there is a following error: 2016-04-29T06:40:57.718177+00:00 notice: (/Stage[main]/Cluster::Mysql/Exec[wait-initial-sync]/returns) ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) 2016-04-29T06:40:57.721064+00:00 err: (/Stage[main]/Cluster::Mysql/Exec[wait-initial-sync]) Failed to call refresh: mysql -uclustercheck -pnOTvDD8mPReXplHCuDPcUI2X -Nbe "show status like 'wsrep_local_state_comment'" | grep -q -e Synced && sleep 10 returned 1 instead of one of [0] There are also lots of "32 (broken pipe)" errors in mysqld logs. MOS 9.0 systest run 661 on iso 201 (https://packaging-ci.infra.mirantis.net/job/9.0-pkg-systest-ubuntu/661/console) failed with the following traceback: FAIL: Deploy ceph HA with RadosGW for objects ---------------------------------------------------------------------- Traceback (most recent call last):   File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func     compatability.capture_type_error(s_func)   File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error     func()   File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func     func(test_case.state.get_state())   File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 120, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/tests/test_ceph.py", line 502, in ceph_rados_gw     self.fuel_web.deploy_cluster_wait(cluster_id)   File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 455, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 440, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 491, in wrapper     return func(*args, **kwargs)   File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 498, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/helpers/decorators.py", line 382, in wrapper     return func(*args, **kwargs)   File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 817, in deploy_cluster_wait     self.assert_task_success(task, interval=interval, timeout=timeout)   File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/__init__.py", line 59, in wrapped     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.0-pkg-systest-ubuntu/fuel-qa/fuelweb_test/models/fuel_web_client.py", line 319, in assert_task_success     task["name"], task['status'], 'ready', _message(task) AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[database/3], Task[database/2] Stopping the deployment process!' In the remote/puppet-apply.log logs on node 3 there are a couple of failures: 2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) Fetched 8564 kB in 1s (8512 kB/s) 2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) dpkg: error: dpkg status database is locked by another process 2016-04-29T06:26:39.718030+00:00 err: (Package[pacemaker](provider=apt_fuel)) E: Sub-process /usr/bin/dpkg returned an error code (2) 2016-04-29T06:26:39.718277+00:00 err: (Package[pacemaker](provider=apt_fuel)) Fetched 8564 kB in 1s (8512 kB/s) 2016-04-29T06:26:39.718409+00:00 err: (Package[pacemaker](provider=apt_fuel)) dpkg: error: dpkg status database is locked by another process 2016-04-29T06:26:39.718409+00:00 err: (Package[pacemaker](provider=apt_fuel)) E: Sub-process /usr/bin/dpkg returned an error code (2) and 2016-04-29T06:36:44.766309+00:00 err: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES) On node 2 there is a following error: 2016-04-29T06:40:57.718177+00:00 notice: (/Stage[main]/Cluster::Mysql/Exec[wait-initial-sync]/returns) ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) 2016-04-29T06:40:57.721064+00:00 err: (/Stage[main]/Cluster::Mysql/Exec[wait-initial-sync]) Failed to call refresh: mysql -uclustercheck -pnOTvDD8mPReXplHCuDPcUI2X -Nbe "show status like 'wsrep_local_state_comment'" | grep -q -e Synced && sleep 10 returned 1 instead of one of [0] There are also lots of "32 (broken pipe)" errors in mysqld logs and these: 2016-04-29T14:25:07.973435+00:00 err: 2016-04-29 14:25:08 32501 [ERROR] WSREP: Failed to read from: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.109.21.4:4444/xtrabackup_sst' --auth 'root:x8LrXWoKZMUfBtuQy3PnJ7qB' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '6864c121-0e13-11e6-b5b6-a3c80f4596ac:0' 2016-04-29T14:25:07.973936+00:00 err: 2016-04-29 14:25:08 32501 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.109.21.4:4444/xtrabackup_sst' --auth 'root:x8LrXWoKZMUfBtuQy3PnJ7qB' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '6864c121-0e13-11e6-b5b6-a3c80f4596ac:0': 22 (Invalid argument) 2016-04-29T14:25:07.973936+00:00 err: 2016-04-29 14:25:08 32501 [ERROR] WSREP: Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.109.21.4:4444/xtrabackup_sst' --auth 'root:x8LrXWoKZMUfBtuQy3PnJ7qB' --socket '/var/run/mysqld/mysqld.sock' --datadir '/var/lib/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' '' --gtid '6864c121-0e13-11e6-b5b6-a3c80f4596ac:0'
2016-05-04 10:26:15 Timur Nurlygayanov marked as duplicate 1571763