1. Start Deployment of any env (I used Neutron VLAN in HA, only 3 controllers, no computes)
2. Turn off one node when provisioning is finished
3. During the deployment process of other nodes, click on Stop Deployment button.
You will observe debug messages in Astute log:
2014-02-23 20:09:11 DEBUG [10800] Retry #1 to run mcollective agent on nodes: '1'
After 5 retries it fails with:
2014-02-23 20:12:42 ERR
[10800] Error running RPC method stop_deploy_task: 3142e9b5-c981-48c7-856b-db3752e911c9: MCollective agents '1' didn't respond within the allotted time.
, trace: ["/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/astute-0.0.2/lib/astute/mclient.rb:113:in `check_results_with_retries'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/astute-0.0.2/lib/astute/mclient.rb:61:in `method_missing'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/astute-0.0.2/lib/astute/orchestrator.rb:162:in `stop_puppet_deploy'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/dispatcher.rb:187:in `stop_current_task'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/dispatcher.rb:156:in `stop_deploy_task'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:132:in `dispatch_message'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:85:in `block in dispatch'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:83:in `each'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:83:in `each_with_index'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:83:in `dispatch'", "/opt/rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/naily-0.1.0/lib/naily/server.rb:78:in `block in perform_service_job'"]
2014-02-23 20:12:42 ERR
[10800] MCollective agents '1' didn't respond within the allotted time.
Expected behavior is that we can stop deployment or provisioning even if there is any disaster with one of a few nodes, this is actually the basic use case of the feature. You stop deployment, then bring up failing node back, and continue deployment.
I was also observing in logs, there are a few pretty strange things, and these needs to be reported as separated bugs:
* some logs lines are duplicated, one is normal Ruby hash, another is JSON format of it. Low priority issue to fix
* Stop deployment task, looks like, didn't actually break initial deployment task, as there was continuing log with an attempt to run puppet on one of the nodes. Puppet was previously killed on that node and deployment failed, as there was a lockfile. Do we really kill that thread (actually, that's fork.... so I don't know how we approach it at the moment).
Env: {"build_id": "2014-02-23_01-17-30", "mirantis": "yes", "build_number": "180", "nailgun_sha": "f786786894acc331a4b53b31f33e373ef95ccdfc", "ostf_sha": "b8f16a0288cbf39e11e0b4a41a3f63e6b87dcc4b", "fuelmain_sha": "421f2aaa7b1494e899368908b295acc5fe7e012f", "astute_sha": "3d43abeefb60677ce6cae83d31ebbba1ff3cdbe2", "release": "4.1", "fuellib_sha": "e3ea44c3b607f37401a268a91956c9d222a81bab"}
Related fix proposed to branch: master
Review: https:/