Removing node from cluster after upgrade failed with Error running RPC method remove_nodes: undefined method `empty?' for nil:NilClass

[Andrey Sledzinskiy]

Removing node from cluster after upgrade failed with Error running RPC method remove_nodes: undefined method `empty?' for nil:NilClass

Steps:
1. Create and deploy next 6.1 cluster - CentOS, Neutron Vlan, Ceph for volumes and images, 1 controller+ceph, 2 compute+ceph
2. After deployment upgrade master to 7.0
3. Delete 1 compute+ceph node
4. Start re-deployment

Actual result - deployment failed:
2015-09-07T06:09:50 err: [636] Error running RPC method remove_nodes: undefined method `empty?' for nil:NilClass, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/cobbler_manager.rb:78:in `ensure in remove_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/cobbler_manager.rb:83:in `remove_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/provision.rb:187:in `remove_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/orchestrator.rb:112:in `remove_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/dispatcher.rb:160:in `remove_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:146:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:107:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:64:in `call'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:64:in `block in each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/task_queue.rb:56:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:105:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:105:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:89:in `block in perform_main_job'"]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "284"
  build_id: "284"
  nailgun_sha: "5c33995a2e6d9b1b8cdddfa2630689da5084506f"
  python-fuelclient_sha: "1ce8ecd8beb640f2f62f73435f4e18d1469979ac"
  fuel-agent_sha: "082a47bf014002e515001be05f99040437281a2d"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "8283dc2932c24caab852ae9de15f94605cc350c6"
  fuel-library_sha: "f81fdabe6c05be7a3d11d88a7c3a8f3931921c73"
  fuel-ostf_sha: "1f08e6e71021179b9881a824d9c999957fcc7045"
  fuelmain_sha: "9ab01caf960013dc882825dc9b0e11ccf0b81cb0"

[Ihor Kalnytskyi (ikalnytskyi)]

Raise to Critical because it affects not only upgrade case, but all "node removal" cases.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/220974

[Anastasia Palkina (apalkina)]

I have a similar error for stopping deployment:

2015-09-07 14:12:42 ERR [609] Error running RPC method stop_deploy_task: undefined method `empty?' for nil:NilClass, trace:
["/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/cobbler_manager.rb:78:in `ensure in remove_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/cobbler_manager.rb:83:in `remove_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/provision.rb:187:in `remove_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/orchestrator.rb:112:in `remove_nodes'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/dispatcher.rb:242:in `stop_current_task'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/dispatcher.rb:209:in `stop_deploy_task'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:148:in `dispatch_message'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:107:in `block in dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:105:in `each'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:105:in `each_with_index'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:105:in `dispatch'",
 "/usr/lib64/ruby/gems/2.1.0/gems/astute-7.0.0/lib/astute/server/server.rb:100:in `block in perform_service_job'"]

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/220974
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=14f51fad86a181bac46cced8832c0b49d6a7dd9a
Submitter: Jenkins
Branch: master

commit 14f51fad86a181bac46cced8832c0b49d6a7dd9a
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Sep 3 17:58:23 2015 +0300

    Remove nodes by MACs only during provisioning

    The thing is nodes when being provisioned have
    'interfaces' field which contains all the information
    about network interfaces. This field is formatted here
    nailgun/nailgun/orchestrator/provisioning_serializers.py:serialize_interfaces
    and we can use this field to find system in the Cobbler
    with the same MAC addresses if there exist any.
    However, deletion task contains nodes which
    do not have 'interfaces' field. So, we can try to
    find nodes by their MACs only during provisioning,
    not during deletion.

    Besides, this original bug #1491725 used to appear
    only during provisioning and this MAC based hack should
    only be used where it is really needed.

    Change-Id: I05fcbb28eee5cbfc5260e3b2a4b423548952a4dd
    Closes-Bug: #1492974

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/221069

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/221069
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=a717657232721a7fafc67ff5e1c696c9dbeb0b95
Submitter: Jenkins
Branch: stable/7.0

commit a717657232721a7fafc67ff5e1c696c9dbeb0b95
Author: Vladimir Kozhukalov <email address hidden>
Date: Thu Sep 3 17:58:23 2015 +0300

    Remove nodes by MACs only during provisioning

    The thing is nodes when being provisioned have
    'interfaces' field which contains all the information
    about network interfaces. This field is formatted here
    nailgun/nailgun/orchestrator/provisioning_serializers.py:serialize_interfaces
    and we can use this field to find system in the Cobbler
    with the same MAC addresses if there exist any.
    However, deletion task contains nodes which
    do not have 'interfaces' field. So, we can try to
    find nodes by their MACs only during provisioning,
    not during deletion.

    Besides, this original bug #1491725 used to appear
    only during provisioning and this MAC based hack should
    only be used where it is really needed.

    Change-Id: I05fcbb28eee5cbfc5260e3b2a4b423548952a4dd
    Closes-Bug: #1492974

[Andrey Sledzinskiy (asledzinskiy)]

мукшашув щт {

    "build_id": "287",
    "build_number": "287",
    "release_versions":

{

    "2015.1.0-7.0":

{

    "VERSION":

{

    "build_id": "287",
    "build_number": "287",
    "api": "1.0",
    "fuel-library_sha": "43224223dab8cf9627b5ecf737e60216a3fdd114",
    "nailgun_sha": "46a7a2177a0b7ef91422284c1c90295fee8f5c84",
    "feature_groups":

            [
                "mirantis"
            ],
            "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
            "openstack_version": "2015.1.0-7.0",
            "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d",
            "production": "docker",
            "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac",
            "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95",
            "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045",
            "release": "7.0",
            "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "43224223dab8cf9627b5ecf737e60216a3fdd114",
"nailgun_sha": "46a7a2177a0b7ef91422284c1c90295fee8f5c84",
"feature_groups":

    [
        "mirantis"
    ],
    "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
    "openstack_version": "2015.1.0-7.0",
    "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d",
    "production": "docker",
    "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac",
    "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95",
    "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045",
    "release": "7.0",
    "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"

}

[Nastya Urlapova (aurlapova)]

@Andrey, I guess you want to say - verified on ...
:)
Thanks for your hard work!