Activity log for bug #1648757

Date Who What changed Old value New value Message
2016-12-09 10:15:28 Elena Ezhova bug added bug
2016-12-09 10:16:34 Elena Ezhova attachment added fail_error_ceph_rados_gw-fuel-snapshot-2016-12-09_09-00-06.tar https://bugs.launchpad.net/fuel/+bug/1648757/+attachment/4789436/+files/fail_error_ceph_rados_gw-fuel-snapshot-2016-12-09_09-00-06.tar
2016-12-09 10:23:52 Elena Ezhova tags bvt-failure
2016-12-09 10:34:55 Elena Ezhova description 9.x BVT job [0] failed with the following trace: 2016-12-09 09:02:21 - ERROR decorators.py:126 -- Traceback (most recent call last): File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 120, in wrapper result = func(*args, **kwargs) File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 507, in ceph_rados_gw self.fuel_web.deploy_cluster_wait(cluster_id) File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 460, in wrapper result = func(*args, **kwargs) File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 445, in wrapper result = func(*args, **kwargs) File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 496, in wrapper return func(*args, **kwargs) File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 503, in wrapper result = func(*args, **kwargs) File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 387, in wrapper return func(*args, **kwargs) File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 953, in deploy_cluster_wait self.assert_task_success(task, interval=interval, timeout=timeout) File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 215, in wrapped result = func(*args, **kwargs) File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 354, in assert_task_success task["name"], task['status'], 'ready', _message(task) File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal raise ASSERTION_ERROR(message) AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[tools/1], Task[tools/3], Task[tools/2], Task[tools/5], Task[tools/4], Task[tools/6] Stopping the deployment process!' Errors in astute log are the same for each of the nodes: 2016-12-09 08:59:59 ERROR [15379] Node 4, task tools, manifest /etc/puppet/modules/osnailyfacter/modular/tools/tools.pp, status: stopped 2016-12-09 08:59:59 ERROR [15379] Node has failed to deploy. There is no more retries for puppet run. Node 4, task tools, manifest /etc/puppet/modules/osnailyfacter/modular/tools/tools.pp Errors in puppet-apply.log on nodes [1] show that execution of `/bin/systemctl enable atop` and `/bin/systemctl enable postfix` fail with return code 1: `Failed to issue method call: No such file or directory`. [0] https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/614 [1] https://paste.mirantis.net/show/2847/ 9.x BVT jobs [0], [1] failed with the following trace: 2016-12-09 09:02:21 - ERROR decorators.py:126 -- Traceback (most recent call last):   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 120, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 507, in ceph_rados_gw     self.fuel_web.deploy_cluster_wait(cluster_id)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 460, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 445, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 496, in wrapper     return func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 503, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 387, in wrapper     return func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 953, in deploy_cluster_wait     self.assert_task_success(task, interval=interval, timeout=timeout)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 215, in wrapped     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 354, in assert_task_success     task["name"], task['status'], 'ready', _message(task)   File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal     raise ASSERTION_ERROR(message) AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[tools/1], Task[tools/3], Task[tools/2], Task[tools/5], Task[tools/4], Task[tools/6] Stopping the deployment process!' Errors in astute log are the same for each of the nodes: 2016-12-09 08:59:59 ERROR [15379] Node 4, task tools, manifest /etc/puppet/modules/osnailyfacter/modular/tools/tools.pp, status: stopped 2016-12-09 08:59:59 ERROR [15379] Node has failed to deploy. There is no more retries for puppet run. Node 4, task tools, manifest /etc/puppet/modules/osnailyfacter/modular/tools/tools.pp Errors in puppet-apply.log on nodes [2] show that execution of `/bin/systemctl enable atop` and `/bin/systemctl enable postfix` fail with return code 1: `Failed to issue method call: No such file or directory`. [0] https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/614 [1] https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/613/ [2] https://paste.mirantis.net/show/2847/
2016-12-09 13:36:34 Alexander Ignatov fuel: status New Confirmed
2016-12-09 13:36:50 Alexander Ignatov fuel: assignee Fuel Sustaining (fuel-sustaining-team)
2016-12-09 13:36:52 Alexander Ignatov fuel: importance Undecided High
2016-12-09 13:36:57 Alexander Ignatov fuel: milestone 9.2
2016-12-09 13:39:22 Elena Ezhova description 9.x BVT jobs [0], [1] failed with the following trace: 2016-12-09 09:02:21 - ERROR decorators.py:126 -- Traceback (most recent call last):   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 120, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 507, in ceph_rados_gw     self.fuel_web.deploy_cluster_wait(cluster_id)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 460, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 445, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 496, in wrapper     return func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 503, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 387, in wrapper     return func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 953, in deploy_cluster_wait     self.assert_task_success(task, interval=interval, timeout=timeout)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 215, in wrapped     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 354, in assert_task_success     task["name"], task['status'], 'ready', _message(task)   File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal     raise ASSERTION_ERROR(message) AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[tools/1], Task[tools/3], Task[tools/2], Task[tools/5], Task[tools/4], Task[tools/6] Stopping the deployment process!' Errors in astute log are the same for each of the nodes: 2016-12-09 08:59:59 ERROR [15379] Node 4, task tools, manifest /etc/puppet/modules/osnailyfacter/modular/tools/tools.pp, status: stopped 2016-12-09 08:59:59 ERROR [15379] Node has failed to deploy. There is no more retries for puppet run. Node 4, task tools, manifest /etc/puppet/modules/osnailyfacter/modular/tools/tools.pp Errors in puppet-apply.log on nodes [2] show that execution of `/bin/systemctl enable atop` and `/bin/systemctl enable postfix` fail with return code 1: `Failed to issue method call: No such file or directory`. [0] https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/614 [1] https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/613/ [2] https://paste.mirantis.net/show/2847/ 9.x BVT jobs [0], [1], [2] failed with the following trace: 2016-12-09 09:02:21 - ERROR decorators.py:126 -- Traceback (most recent call last):   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 120, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 507, in ceph_rados_gw     self.fuel_web.deploy_cluster_wait(cluster_id)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 460, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 445, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 496, in wrapper     return func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 503, in wrapper     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 387, in wrapper     return func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 953, in deploy_cluster_wait     self.assert_task_success(task, interval=interval, timeout=timeout)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/core/helpers/log_helpers.py", line 215, in wrapped     result = func(*args, **kwargs)   File "/home/jenkins/workspace/9.x.main.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 354, in assert_task_success     task["name"], task['status'], 'ready', _message(task)   File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 55, in assert_equal     raise ASSERTION_ERROR(message) AssertionError: Task 'deploy' has incorrect status. error != ready, 'Deployment has failed. All nodes are finished. Failed tasks: Task[tools/1], Task[tools/3], Task[tools/2], Task[tools/5], Task[tools/4], Task[tools/6] Stopping the deployment process!' Errors in astute log are the same for each of the nodes: 2016-12-09 08:59:59 ERROR [15379] Node 4, task tools, manifest /etc/puppet/modules/osnailyfacter/modular/tools/tools.pp, status: stopped 2016-12-09 08:59:59 ERROR [15379] Node has failed to deploy. There is no more retries for puppet run. Node 4, task tools, manifest /etc/puppet/modules/osnailyfacter/modular/tools/tools.pp Errors in puppet-apply.log on nodes [3] show that execution of `/bin/systemctl enable atop` and `/bin/systemctl enable postfix` fail with return code 1: `Failed to issue method call: No such file or directory`. [0] https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/614 [1] https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/613/ [2] https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/615/ [3] https://paste.mirantis.net/show/2847/
2016-12-09 13:48:19 Stanislaw Bogatkin fuel: assignee Fuel Sustaining (fuel-sustaining-team) Stanislaw Bogatkin (sbogatkin)
2016-12-09 15:38:07 Victor Ryzhenkin fuel: importance High Critical
2016-12-09 15:59:22 Stanislaw Bogatkin marked as duplicate 1648732