Environments are not removed, stay in state 'Removing'

[Anastasia Palkina]

"build_id": "2015-02-07_22-55-01",
"ostf_sha": "6c046b69d29021524906109f18092363505ee222",
"build_number": "105",
"release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-02-07_22-55-01", "ostf_sha": "6c046b69d29021524906109f18092363505ee222", "build_number": "105", "api": "1.0", "nailgun_sha": "6d1769b21819f8fb4195f1bd9c44c038721ae3d4", "production": "docker", "python-fuelclient_sha": "521c2491f7f04f31d8c85db68499cd193d4904e3", "astute_sha": "7e6e6f9188bd69c603853b10d4a55149363323cc", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "", "fuellib_sha": "769af7fe30225cd15638ea2e6dffaa286bc06da1"}}},
"auth_required": true,
"api": "1.0",
"nailgun_sha": "6d1769b21819f8fb4195f1bd9c44c038721ae3d4",
"production": "docker",
"python-fuelclient_sha": "521c2491f7f04f31d8c85db68499cd193d4904e3",
"astute_sha": "7e6e6f9188bd69c603853b10d4a55149363323cc",
"feature_groups": ["mirantis"],
"release": "6.1",
"fuelmain_sha": "",
"fuellib_sha": "769af7fe30225cd15638ea2e6dffaa286bc06da1"

1. Enable experimental mode
2. Create new environment (Ubuntu, HA mode)
3. Choose nova-network, flat
4. Add 1 controller, 1 compute, 1 zabbix
5. Choose image-based provisioning
6. Start deployment. It has failed because zabbix is not running for granular deployment
7. Try to delete environment. It is not removed, stays in state 'Removing'
Nodes in state 'Pending deletion' (see screen). Thy are not bootstraped

[Dmitry Ilyin (idv1985)]

Yes, I've seen this situation several times too. Had to clean out tasks table of the Nailgun db.
I guess it can happen if Astute was restarted during the task.

[Vladimir Sharshov (vsharshov)]

Looks like problem with Zabbix and more commonly with condition. We does not need to perform request to Zabbix in case of deleting entire cluster.

2015-02-09 11:52:01.335 DEBUG [7f594b058740] (cluster) Trying to execute cluster deletion task
2015-02-09 11:52:01.354 DEBUG [7f594b058740] (manager) Removing cluster tasks
2015-02-09 11:52:01.355 DEBUG [7f594b058740] (manager) Labeling cluster nodes to delete
2015-02-09 11:52:01.370 DEBUG [7f594b058740] (manager) Creating cluster deletion task
2015-02-09 11:52:01.385 DEBUG [7f594b058740] (task) Cluster deletion task is running
2015-02-09 11:52:01.386 DEBUG [7f594b058740] (task) DeletionTask.execute(task=650e1e3e-2fa0-4813-981d-7e42a257b550)
2015-02-09 11:52:01.386 DEBUG [7f594b058740] (task) Nodes deletion task is running
2015-02-09 11:52:01.422 DEBUG [7f594b058740] (task) Removing nodes [{'slave_name': u'node-1', 'id': 1, 'roles': [u'controller'], 'uid': 1}, {'slave_name': u'node-2', 'id': 2, 'roles': [u'compute'], 'uid': 2}, {'slave_name': u'node-3', 'id': 3, 'roles': [u'zabbix-server'], 'uid': 3}] from zabbix
2015-02-09 11:52:01.423 DEBUG [7f594b058740] (zabbix) Zabbix request: {'params': {'password': u'zabbix', 'user': u'admin'}, 'jsonrpc': '2.0', 'id': '1', 'method': 'user.authenticate'}
2015-02-09 11:52:01.477 DEBUG [7f594b058740] (zabbix) Zabbix response: {'jsonrpc': '2.0', 'id': '1', 'error': {'message': "Error connecting to database [Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)]", 'code': 1}}
2015-02-09 11:52:01.497 ERROR [7f594b058740] (manager) Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nailgun/task/manager.py", line 55, in _call_silently
    to_return = method(task, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nailgun/task/task.py", line 468, in execute
    DeletionTask.execute(task, 'remove_cluster_resp')
  File "/usr/lib/python2.6/site-packages/nailgun/task/task.py", line 319, in execute
    zabbix_credentials, nodes_to_delete
  File "/usr/lib/python2.6/site-packages/nailgun/utils/zabbix.py", line 86, in remove_from_zabbix
    auth = cls._zabbix_auth(url, user, password)
  File "/usr/lib/python2.6/site-packages/nailgun/utils/zabbix.py", line 66, in _zabbix_auth
    auth_hash = cls._make_zabbix_request(url, method, params)
  File "/usr/lib/python2.6/site-packages/nailgun/utils/zabbix.py", line 52, in _make_zabbix_request
    data = result['error']['data']
KeyError: 'data'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/155696

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/155696
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=2eee83b38df58e06fa297994efa5f6f19ce65f0a
Submitter: Jenkins
Branch: master

commit 2eee83b38df58e06fa297994efa5f6f19ce65f0a
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Feb 13 13:19:07 2015 +0300

    Do not call Zabbix in case of deletion of cluster

    No reason to call Zabbix to report about changes because
    we delete node with it few seconds after.

    Also fixed:

    zabbix error respond may not contain data field

    Example:

    { 'jsonrpc': '2.0', 'id': '1', 'error': {
      'message': "Error connecting to database [Can't connect
                  to local MySQL server through socket
                  '/var/run/mysqld/mysqld.sock' (2)]",
      'code': 1}}

    Change-Id: Ia25c7107b3dc535b6e14db74a82d096fd6caa4d0
    Closes-Bug: #1419775

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/161241

[Anastasia Palkina (apalkina)]

Verified on ISO #165

"build_id": "2015-03-05_22-54-44", "ostf_sha": "8df5f2fcdae3bc9ea7d700ffd64db820baf51914", "build_number": "165", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-05_22-54-44", "ostf_sha": "8df5f2fcdae3bc9ea7d700ffd64db820baf51914", "build_number": "165", "api": "1.0", "nailgun_sha": "f12221d79e0d97c7b4405331e11a54fc5dcfcd4e", "production": "docker", "python-fuelclient_sha": "4eb787f1ad969bd23c93d192865543dbd45a8626", "astute_sha": "ca7635a356a90404d3dedb5cf26f1d16e07144a9", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0e45b31db1677651d6ddb1c852d62ebfd8875dcd", "fuellib_sha": "07288d7bfde840b7ec47292ff96a3b670a79c859"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "f12221d79e0d97c7b4405331e11a54fc5dcfcd4e", "production": "docker", "python-fuelclient_sha": "4eb787f1ad969bd23c93d192865543dbd45a8626", "astute_sha": "ca7635a356a90404d3dedb5cf26f1d16e07144a9", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "0e45b31db1677651d6ddb1c852d62ebfd8875dcd", "fuellib_sha": "07288d7bfde840b7ec47292ff96a3b670a79c859"

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/161241
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=4e0480f345fd5abb4542c2b2143c0489a00d2af8
Submitter: Jenkins
Branch: stable/6.0

commit 4e0480f345fd5abb4542c2b2143c0489a00d2af8
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Fri Feb 13 13:19:07 2015 +0300

    Do not call Zabbix in case of deletion of cluster

    No reason to call Zabbix to report about changes because
    we delete node with it few seconds after.

    Also fixed:

    zabbix error respond may not contain data field

    Example:

    { 'jsonrpc': '2.0', 'id': '1', 'error': {
      'message': "Error connecting to database [Can't connect
                  to local MySQL server through socket
                  '/var/run/mysqld/mysqld.sock' (2)]",
      'code': 1}}

    Change-Id: Ia25c7107b3dc535b6e14db74a82d096fd6caa4d0
    Closes-Bug: #1419775
    (cherry picked from commit 2eee83b38df58e06fa297994efa5f6f19ce65f0a)