Failed to delete controller+ironic node

[Kyrylo Romanenko]

Steps:
1. Deploy cluster:

1 Controller
1 Compute
1 Ironic.

Settings:
Compute - QEMU.
Network - Neutron with VLAN segmentation, DVR enabled.
Storage Backends - Cinder LVM over iSCSI for volumes.

2. Add 1 Ironic and 1 Controller+Ironic node.
3. Deploy changes.
4. Run OSTF. Run happy-path to boot Ironic VM instance.
5. Delete 1 Ironic and 1 Controller+Ironic node.
6. Deploy changes.
Changes deployment failed after several minutes.
Astute log errors:
http://paste.openstack.org/show/484418/

Note: separate Ironic nodes can be deleted successfully.
Possibly bug consists in deletion of Controller+Ironic node or both.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "429"
  build_id: "429"
  fuel-nailgun_sha: "12b15b2351e250af41cc0b10d63a50c198fe77d8"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "3eaf4f4a9b88b287a10cc19e9ce6a62298cc4013"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "85de57080a18fda18e5325f06eaf654b1b931592"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "e8e36cff332644576d7853c80b8a53d5b955420a"

[Kyrylo Romanenko (kromanenko)]

Diagnostic snapshot:
https://drive.google.com/a/mirantis.com/file/d/0B6E70aHvCcRQMDJLN1B4dlducFU/view?usp=sharing

[Kyrylo Romanenko (kromanenko)]

1. Reproduced it one more time:

1 Controller
1 Compute
1 Ironic
1 Ironic+Compute
Settings:
Compute - QEMU.
Network - Neutron with VLAN segmentation.
Storage Backends - Cinder LVM over iSCSI for volumes.

2. Delete 1 Ironic+Compute node.
3. Deploy changes. Get error:

 [85] Error running RPC method granular_deploy: Missing a required parameter uids, trace:
["/usr/share/gems/gems/astute-8.0.0/lib/astute/nailgun_hooks.rb:312:in `validate_presence'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/nailgun_hooks.rb:88:in `copy_files_hook'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/nailgun_hooks.rb:32:in `block in process'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/nailgun_hooks.rb:26:in `each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/nailgun_hooks.rb:26:in `process'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/deployment_engine/granular_deployment.rb:215:in `pre_deployment_actions'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/deployment_engine.rb:36:in `deploy'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/orchestrator.rb:216:in `deploy_cluster'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/orchestrator.rb:52:in `granular_deploy'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/dispatcher.rb:92:in `granular_deploy'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:189:in `dispatch_message'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:146:in `block in dispatch'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `each_with_index'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:144:in `dispatch'",
 "/usr/share/gems/gems/astute-8.0.0/lib/astute/server/server.rb:123:in `block in perform_main_job'"]

Same ISO
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "429"
  build_id: "429"

[Roman Podoliaka (rpodolyaka)]

Dmitry, tag is supposed to denote the problematic area in the code, not the assignee team.

[Andrey Shestakov (ashestakov)]

fix proposed https://review.openstack.org/#/c/274026

[Roman Podoliaka (rpodolyaka)]

Fix merged : https://review.openstack.org/#/q/Icb6c3cc01c194a60bc5324a963e2f30e149b4bcf,n,z

[Andrey Lavrentyev (alavrentyev)]

Verified on ISO #567

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "567"
  build_id: "567"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "33634ec27be77ecfb0b56b7e07497ad86d1fdcd3"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

[Sofiia Andriichenko (sandriichenko)]

Verified
cat /etc/fuel_build_id:
 197
cat /etc/fuel_build_number:
 197
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0
rpm -qa | egrep 'fuel|astute|network-checker|nailgun|packetary|shotgun':
 fuel-release-9.0.0-1.mos6332.noarch
 fuel-misc-9.0.0-1.mos8290.noarch
 python-packetary-9.0.0-1.mos131.noarch
 fuel-openstack-metadata-9.0.0-1.mos8648.noarch
 fuel-provisioning-scripts-9.0.0-1.mos8648.noarch
 python-fuelclient-9.0.0-1.mos306.noarch
 fuel-9.0.0-1.mos6332.noarch
 fuel-nailgun-9.0.0-1.mos8648.noarch
 rubygem-astute-9.0.0-1.mos738.noarch
 fuel-library9.0-9.0.0-1.mos8290.noarch
 fuel-agent-9.0.0-1.mos272.noarch
 fuel-ui-9.0.0-1.mos2658.noarch
 fuel-setup-9.0.0-1.mos6332.noarch
 nailgun-mcagents-9.0.0-1.mos738.noarch
 shotgun-9.0.0-1.mos87.noarch
 network-checker-9.0.0-1.mos72.x86_64
 fuel-bootstrap-cli-9.0.0-1.mos272.noarch
 fuel-migrate-9.0.0-1.mos8290.noarch
 fuelmenu-9.0.0-1.mos268.noarch
 fuel-notify-9.0.0-1.mos8290.noarch
 fuel-ostf-9.0.0-1.mos924.noarch
 fuel-mirror-9.0.0-1.mos131.noarch
 fuel-utils-9.0.0-1.mos8290.noarch