MOS Job master.staging.centos.bvt_1 failed with the traceback for master branch

[Timur Nurlygayanov]

The reason of this fail in the following: the deployment of Fuel master node hangs on step with pre-configuration (please see attached screenshot) - and in the result all automated tests/scripts will fail because they can't automatically continue the deployment of Fuel master node.

need to change this behaviour and allow full automated deployment of Fuel node.

Job:
http://jenkins-product.srt.mirantis.net:8080/view/Staging%20Mirrors/job/master.staging.centos.bvt_1/60/console

Traceback:
2015-01-26 15:27:56,987 - INFO environment.py:585 -- Puppet timeout set in 6000.0
ERROR
Prepare master node ... SKIP: Failure in <function setup_master at 0x7ff4927d97d0>
Bootstrap 5 slave nodes ... SKIP: Failure in <function setup_master at 0x7ff4927d97d0>
Deploy cluster in HA mode with Neutron VLAN ... SKIP: Failure in <function setup_master at 0x7ff4927d97d0>

======================================================================
ERROR: Create environment and set up master node
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master.staging.centos.bvt_1/fuelweb_test/tests/base_test_case.py", line 57, in setup_master
    self.env.setup_environment()
  File "/home/jenkins/workspace/master.staging.centos.bvt_1/fuelweb_test/models/environment.py", line 503, in setup_environment
    self.wait_bootstrap()
  File "/home/jenkins/workspace/master.staging.centos.bvt_1/fuelweb_test/models/environment.py", line 597, in wait_bootstrap
    raise Exception('Fuel node deployment failed.')
Exception: Fuel node deployment failed.

[Timur Nurlygayanov (tnurlygayanov)]

The other job:
http://jenkins-product.srt.mirantis.net:8080/view/Staging%20Mirrors/job/master.staging.ubuntu.bvt_2/54/console

failed with the same error.

[Timur Nurlygayanov (tnurlygayanov)]

The issue reproduced manually too - deployment takes long time.

[Mike Scherbakov (mihgen)]

Timur, can you please assign to the appropriate team? For instance, fuel-library if it's deployment issue. Fuel Core Team contains too many people, and I'd like to avoid spamming. Thanks.

[Dmitry Mescheryakov (dmitrymex)]

Raising to critical since that happened on test.staging.mirror job

[Igor Shishkin (teran)]

Timur,

description like 'It was failed with message' sounds awesome but can't give us idea where dig.
Please investigate what happened there and assign this issue to proper team.

[Timur Nurlygayanov (tnurlygayanov)]

Hi,

the reason of this fail in the following: the deployment of Fuel master node hangs on step with pre-configuration (please see attached screenshot) - and in the result all automated tests/scripts will fail because they can't automatically continue the deployment of Fuel master node.

need to change this behaviour and allow full automated deployment of Fuel node.

[Timur Nurlygayanov (tnurlygayanov)]

In puppet logs we can see only message that changes were successfully applied.
it is not clear where the root of the issue, looks like the application with the Fuel console configuration tool is broken.

[Matthew Mosesohn (raytrac3r)]

Sorry for the delay in responding to this. Fuelmenu itself is not broken. During system tests it starts up and then is killed with SIGUSR1 signal. That causes it to save and quit cleanly. The only proof we need is that the contents of /etc/fuel/astute.yaml match whatever parameters were used to deploy the test. There is no snapshot to revert here, so we need to reproduce again.

Lastly, Igor Belikov recently made a patch that should make sure system logs are attached when it fails during this step.

[Igor Belikov (ibelikov)]

Yes, https://review.openstack.org/#/c/152605/ should solve attaching logs on master deployment failure.
 AFAIK we're waiting for qa team to move to dedicated fuel-qa repo, after that it would be merged.

[Matthew Mosesohn (raytrac3r)]

Timur do you have a new failure to report? Marking incomplete

[Timur Nurlygayanov (tnurlygayanov)]

Hi Matthew, looks like it was fixed.