Fail to delete primary controller: Method granular_deploy. Deployment failed on nodes 2

[Vladimir]

Steps to reproduce:

1. Deploy cluster: 1 controller, Neutron Vlan, default storages
2. Add 2 controllers
3. Re-deploy cluster
4. Add 2 controllers, 1 compute, 1 cinder node
5. Re-deploy cluster
6. Delete primary controller and add another one controller
7. Re-deploy cluster

Deployment failed with the following error:

Deployment has failed. Method granular_deploy. Deployment failed on nodes 2.
Inspect Astute logs for the details

Astute logs page shows the following message:
Unable to fetch logs.

Error message on the failed node:
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/bin/puppet:8:in `<main>'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet.rb:244:in `override'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
2015-12-12 21:36:01 ERR (/Stage[main]/Rabbitmq::Management/Rabbitmq_user[guest]) /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'

Diagnostic snapshot creation failed: Dump is timed out.

[Vladimir (vushakov)]

Fuel version:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "263"
  build_id: "263"
  fuel-nailgun_sha: "0e09dce510927f2cc490b898e5fe3f813bd791be"
  python-fuelclient_sha: "f033192b84263f0e699458a4274289a5198ae7e4"
  fuel-agent_sha: "660c6514caa8f5fcd482f1cc4008a6028243e009"
  fuel-nailgun-agent_sha: "a33a58d378c117c0f509b0e7badc6f0910364154"
  astute_sha: "48fd58676debcc85951db68df6d77c22daa55e52"
  fuel-library_sha: "ab7e51f345ffb7c256e0f61addcf86553d7c3867"
  fuel-ostf_sha: "23b7ae2a1a57de5a3e1861ffb7805394ca339cc2"
  fuel-mirror_sha: "6534117233a5bdc51d7d47361bc7d511e4b11e6f"
  fuelmenu_sha: "fcb15df4fd1a790b17dd78cf675c11c279040941"
  shotgun_sha: "a0bd06508067935f2ae9be2523ed0d1717b995ce"
  network-checker_sha: "a3534f8885246afb15609c54f91d3b23d599a5b1"
  fuel-upgrade_sha: "1e894e26d4e1423a9b0d66abd6a79505f4175ff6"
  fuelmain_sha: "26adf12c320936a97a9b0a84169a6e58c530e848"

[Ivan Berezovskiy (iberezovskiy)]

Please, provide diagnostic snapshot

[Vladimir (vushakov)]

During original failure diagnostic snapshot creation failed. I will try to reproduce it again and gather it manually.

[Alexander Gubanov (ogubanov)]

I got the same error on MOS 8.0 (build 361) during deploy.
Env: Neutron VLAN - 3 controller nodes, 2 compute with Cinder role

[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "361"
  build_id: "361"
  fuel-nailgun_sha: "53c72a9600158bea873eec2af1322a716e079ea0"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "7463551bc74841d1049869aaee777634fb0e5149"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "ba8063d34ff6419bddf2a82b1de1f37108d96082"
  fuel-ostf_sha: "889ddb0f1a4fa5f839fd4ea0c0017a3c181aa0c1"
  fuel-mirror_sha: "8adb10618bb72bb36bb018386d329b494b036573"
  fuelmenu_sha: "824f6d3ebdc10daf2f7195c82a8ca66da5abee99"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "07d5f1c3e1b352cb713852a3a96022ddb8fe2676"

Error
Deployment has failed. Method granular_deploy. Failed to execute hook 'upload_cirros' command: cd / && ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb

Task: b039f73d-3ad5-4077-bf42-c81ee22d8e0d: shell timeout error: execution expired
Task timeout: 180, Retries: 3

---
uids:
- '1'
parameters:
  retries: 3
  cmd: ruby /etc/puppet/modules/osnailyfacter/modular/astute/upload_cirros.rb
  cwd: /
  timeout: 180
  interval: 20
priority: 900
fail_on_error: true
type: shell
id: upload_cirros
.
Inspect Astute logs for the details

Diagnostic snapshot here https://drive.google.com/a/mirantis.com/file/d/0B_Q36BdjmBtdeENjR0N6dHhidFU/view?usp=sharing

[Roman Podoliaka (rpodolyaka)]

No longer fixing Medium bugs in 8.0

[Ivan Berezovskiy (iberezovskiy)]

Vladimir, please confirm if issue reproduces on the latest 9.0 ISO's.

[Vladimir (vushakov)]

Blocked by: https://bugs.launchpad.net/fuel/+bug/1552153

[Dina Belova (dbelova)]

More than a month in the incomplete state, moving to Invalid. Please move back to Confirmed if the issue is still reproducible.