Astute cannot delete node

[Nastya Urlapova]

VERSION: {
build_id: "2015-05-13_22-55-26",
build_number: "414",
api: "1.0",
fuel-library_sha: "f9310aa52628d70b45a8d9cb735dd309f9b556f8",
nailgun_sha: "afbcea1d4928e588a8858e154770cd9d031a1fc7",
feature_groups: [
"mirantis"
],
openstack_version: "2014.2.2-6.1",
production: "docker",
python-fuelclient_sha: "e4ebbc720c2d5e4546b51758c5796821ed4377f6",
astute_sha: "484d5038b81b7260597be027b1265dc88091026b",
fuel-ostf_sha: "1ae384a28b42d5325bd5d2546aead1e9d502f6a8",
release: "6.1",
fuelmain_sha: "1c03fb884d3e79165a2406d00a5ea8679e35f4f7"

Remove compute from cluster in ha mode with flat nova-network
         Scenario:
            1. Revert "deploy_ha_one_controller_flat" environment
            2. Remove compute node
            3. Deploy changes
            4. Verify node returns to unallocated pull

Deletion failed with error:

[672] Error running RPC method remove_nodes: undefined method `to_set' for [3]:Array, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/pre_delete.rb:137:in `check_for_offline_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:198:in `check_for_offline_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:214:in `perform_pre_deletion_tasks'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/orchestrator.rb:103:in `remove_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/dispatcher.rb:168:in `remove_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:142:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:103:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:101:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-6.1.0/lib/astute/server/server.rb:85:in `block in perform_main_job'"]

[Tatyanka (tatyana-leontovich)]

still can not, 435 iso:
2015-05-19T01:41:15 info: [671] Casting message to Nailgun: {"method"=>"remove_nodes_resp", "args"=>{"task_uuid"=>"d6185da8-e6ee-4bcc-929a-16249ebacc6f", "status"=>"error", "progress"=>100, "error"=>"MCollective is not running on nodes: 2. MCollective must be running to properly delete a node.", "error_nodes"=>[{"uid"=>2}]}}

[Vladimir Sharshov (vsharshov)]

Guys, last problem has been happened because of mcollective. I try to reproduce it on Fuel CI.
First building was freeze after 5 hours of works. It is [0] deletion works great (please do not pay attention to red status. it is red because of problem in Fuel CI itself). I run new test to check it again [1].

Also interesting moment: there is no mcollective log for node '2'. And log from node '3' show that we have problem with connection to server.

We have several bug about mcollective problem, some of them we already fixed:

- https://bugs.launchpad.net/fuel/+bug/1454741
- https://bugs.launchpad.net/fuel/+bug/1449186

If i could not reproduce it on [1], i will close it as incomplete. If you reproduce it again, please provide access to env.

[0] http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/6.1.custom_system_test/119/
[1] http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/6.1.custom_system_test/121/

[Maksym Strukov (unbelll)]

{"build_id": "2015-05-21_04-04-09", "build_number": "446", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}, "2014.2.2-6.1": {"VERSION": {"build_id": "2015-05-21_04-04-09", "build_number": "446", "api": "1.0", "fuel-library_sha": "a03efb582b06bfe8d9776dce244d4a2f2e2ba886", "nailgun_sha": "403c6b7ea3c62bb4fda27eb9cedee37f7144558c", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "e19f1b65792f84c4a18b5a9473f85ef3ba172fce", "astute_sha": "795f8a045400fe82ccc30ae018e85324b3fa1de5", "fuel-ostf_sha": "3dd25a018f2a5c47ec6c885436b3ba69690ef1b9", "release": "6.1", "fuelmain_sha": "5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "a03efb582b06bfe8d9776dce244d4a2f2e2ba886", "nailgun_sha": "403c6b7ea3c62bb4fda27eb9cedee37f7144558c", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "e19f1b65792f84c4a18b5a9473f85ef3ba172fce", "astute_sha": "795f8a045400fe82ccc30ae018e85324b3fa1de5", "fuel-ostf_sha": "3dd25a018f2a5c47ec6c885436b3ba69690ef1b9", "release": "6.1", "fuelmain_sha": "5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93"}

Reproduced on upgraded env (6.0-58 -> 6.1-446)

[Vladimir Sharshov (vsharshov)]

New test with fresh ISO for Ubuntu #446: http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/6.1.custom_system_test/123/

Do not reproduce in http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/6.1.custom_system_test/121/

I will check reproduced on upgraded env

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/184786

[Maksym Strukov (unbelll)]

Cannot reproduce on custom build with included proposed patch
http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/custom_6.1_upgrade/11/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/184786
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=0bd72c72369e743376864e8e8dabfe873d40450a
Submitter: Jenkins
Branch: master

commit 0bd72c72369e743376864e8e8dabfe873d40450a
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Thu May 21 16:23:18 2015 +0300

    Predeletion check: use systemtype agent instead of version.

    The main reason: we never update magents in nodes.
    In this case for old env: 5.1 or 6.0 we could not use
    'version' magent because it is present only in 6.1 nodes.
    But for availabe check we can safely use systemtype
    agent.

    Change-Id: I7c3875d66255997f16b28c10748dfb23b21f5e85
    Closes-Bug: #1455390

[Maksym Strukov (unbelll)]

Verified as fixed on 6.1-451

{"build_id": "2015-05-22_11-53-38", "build_number": "451", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-26_14-25-46", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "58", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "81d38d6f2903b5a8b4bee79ca45a54b76c1361b8", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "fde8ba5e11a1acaf819d402c645c731af450aff0"}}, "2014.2.2-6.1": {"VERSION": {"build_id": "2015-05-22_11-53-38", "build_number": "451", "api": "1.0", "fuel-library_sha": "d2997066fe030983f0625f34fa694fdbbb9845cb", "nailgun_sha": "76441596e4fe6420cc7819427662fa244e150177", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "e19f1b65792f84c4a18b5a9473f85ef3ba172fce", "astute_sha": "0bd72c72369e743376864e8e8dabfe873d40450a", "fuel-ostf_sha": "a4b5864ab99d7f93ed7264ce17d80e7998fda875", "release": "6.1", "fuelmain_sha": "5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "d2997066fe030983f0625f34fa694fdbbb9845cb", "nailgun_sha": "76441596e4fe6420cc7819427662fa244e150177", "feature_groups": ["mirantis"], "openstack_version": "2014.2.2-6.1", "production": "docker", "python-fuelclient_sha": "e19f1b65792f84c4a18b5a9473f85ef3ba172fce", "astute_sha": "0bd72c72369e743376864e8e8dabfe873d40450a", "fuel-ostf_sha": "a4b5864ab99d7f93ed7264ce17d80e7998fda875", "release": "6.1", "fuelmain_sha": "5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93"}