Not enough deployed controllers while there is one controller in env

[Anastasia Kuznetsova]

ENV: Ubuntu, Neutron with VLAN, HA

Steps to reproduce:
1. Prepare the simplest env with no additional services
2. Add 1 controller and 3 compute
3. Start to deploy env
4. Wait for the end of deploy (in my case deploy was failed with Timeout Error, the root of this failure is outdated fuel-main repo, but this failure helped to reproduce this bug)
5. Remove all nodes from env (reset the environment and when nodes will be in 'offline' status, remove each of them by clicking on 'remove' button in the 'Nodes' tab)
6. Try to use this env again: add 1 controller, 3 compute again
7. Deploy it again

Observed result:
Take a look at attached screenshot

{"build_id": "2015-03-12_22-54-44", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "192", "release_versions": {"2014.2-6.1": {"VERSION": {"build_id": "2015-03-12_22-54-44", "ostf_sha": "ecb8e294b0acbdc5b0300d5e39028fb26ecc9088", "build_number": "192", "api": "1.0", "nailgun_sha": "c186f71158ed27b03d8db87561ea66c19e39b452", "production": "docker", "python-fuelclient_sha": "59513d6b75f86060ff5059f39fdd9cca56c83f19", "astute_sha": "ed76b0cacf34a4a683b464ebd86e0beb273b5473", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "3764b8a73b3a93fd7ee66937ba4c4c77da409b78", "fuellib_sha": "fda8128b9ca7a8ce818421040f597a50eece8078"}}}, "auth_required": true, "api": "1.0", "nailgun_sha": "c186f71158ed27b03d8db87561ea66c19e39b452", "production": "docker", "python-fuelclient_sha": "59513d6b75f86060ff5059f39fdd9cca56c83f19", "astute_sha": "ed76b0cacf34a4a683b464ebd86e0beb273b5473", "feature_groups": ["mirantis"], "release": "6.1", "fuelmain_sha": "3764b8a73b3a93fd7ee66937ba4c4c77da409b78", "fuellib_sha": "fda8128b9ca7a8ce818421040f597a50eece8078"}

[Aleksey Kasatkin (alekseyk-ru)]

Caused by this check:

https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/task/task.py#L738-755

Should be fixed to take into account was the environment deployed before error or not.

[Sebastian Kalinowski (prmtl)]

I was trying to reproduce it but with no luck. Let's see if it still occurs.

[Anastasia Kuznetsova (akuznetsova)]

I can't reproduce it too on 323 ISO, seems like this bug was fixed as part of some other

[Sam Stoelinga (sammiestoel)]

Still reproducable see detailed steps here: https://bugs.launchpad.net/fuel/+bug/1465662

Sorry for filing a duplicate

[Igor Zinovik (izinovik)]

Seems that I hitted same problem as described in this bug.

Attaching snapshot, hope it might help to resolve the issue.

master-node# fuel --fuel-version
api: '1.0'
astute_sha: 34e0493afa22999c4a07d3198ceb945116ab7932
auth_required: true
build_id: 2015-07-28_12-51-16
build_number: '103'
feature_groups:
- mirantis
fuel-agent_sha: 2a65f11c10b0aeb5184247635a19740fc3edde21
fuel-library_sha: 8e64ae8e018d5b119c8e2ea49f6a83467b970a66
fuel-nailgun-agent_sha: 1512b9af6b41cc95c4d891c593aeebe0faca5a63
fuel-ostf_sha: 9e32b35e9a774eec41b86cd5a6c63b71a3efa4bd
fuelmain_sha: 0d6ff4c9ec156196cc4d9a36db314ba7346942f7
nailgun_sha: d7fe1047caeb4503970c7d39689e133b28b85b22
openstack_version: 2015.1.0-7.0
production: docker
python-fuelclient_sha: f04e6c46783ecd6000df31b61b6749da66d4d828
release: '7.0'
release_versions:
  2015.1.0-7.0:
    VERSION:
      api: '1.0'
      astute_sha: 34e0493afa22999c4a07d3198ceb945116ab7932
      build_id: 2015-07-28_12-51-16
      build_number: '103'
      feature_groups:
      - mirantis
      fuel-agent_sha: 2a65f11c10b0aeb5184247635a19740fc3edde21
      fuel-library_sha: 8e64ae8e018d5b119c8e2ea49f6a83467b970a66
      fuel-nailgun-agent_sha: 1512b9af6b41cc95c4d891c593aeebe0faca5a63
      fuel-ostf_sha: 9e32b35e9a774eec41b86cd5a6c63b71a3efa4bd
      fuelmain_sha: 0d6ff4c9ec156196cc4d9a36db314ba7346942f7
      nailgun_sha: d7fe1047caeb4503970c7d39689e133b28b85b22
      openstack_version: 2015.1.0-7.0
      production: docker
      python-fuelclient_sha: f04e6c46783ecd6000df31b61b6749da66d4d828
      release: '7.0'

[Artem Roma (aroma-x)]

Have not managed to reproduce the issue by myself. Also, based on analyzing of all attached snapshot logs and db dumps, it seems reset environment action has not been performed, since there are no evidences of such in any logs whatsoever. Could someone confirm that the environment was reset successfully? Also, Is the bug still being reproduced? In the case, please provide fresh diagnostic info, yet better access to actual environment. Till the confirmation move this issue to incomplete status.

[Anastasia Kuznetsova (akuznetsova)]

Bug was reproduces again.

Build info:

[root@nailgun ~]# cat /etc/nailgun/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "156"
  build_id: "2015-08-10_17-24-26"
  nailgun_sha: "9a6ac9b08733ded67bc65345a37787886ff4249d"
  python-fuelclient_sha: "ec5c02b3848fe1d15e5b50b323c3dda030f8e1ac"
  fuel-agent_sha: "57145b1d8804389304cd04322ba0fb3dc9d30327"
  fuel-nailgun-agent_sha: "e01693992d7a0304d926b922b43f3b747c35964c"
  astute_sha: "e1d3a435e5df5b40cbfb1a3acf80b4176d15a2dc"
  fuel-library_sha: "c4b2b9a1b464d7f538635cbfc9304a714ededc41"
  fuel-ostf_sha: "c7f745431aa3c147f2491c865e029e0ffea91c47"
  fuelmain_sha: "826387d1a05a40e4849433598442024bb1e3b7c7"

Steps to reproduce:
1. Create new fuel environment:Ubuntu, HA, Neutron XVLAN, 3 controllers (in my case env was with Ceilo, so all controllers have mongo role too), 1 compute
2. Deploy it (deployment was failed because of bug in Ceilo)
3. After deployment I reseted environment
4. Only after that I manually removed all nodes from it
5. When all nodes were discovered again, I added them to the env (same configuration: 3 controllers, 1 compute)
5. Click on deploy

Observed result:
Error
Not enough deployed controllers - deployed cluster requires at least 1 deployed controller.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/212053

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/212053
Committed: https://git.openstack.org/cgit/stackforge/fuel-web/commit/?id=8943ada643b37361910187ffddebe6ff35860ebc
Submitter: Jenkins
Branch: master

commit 8943ada643b37361910187ffddebe6ff35860ebc
Author: Artem Roma <email address hidden>
Date: Wed Aug 12 17:40:14 2015 +0300

    Add check to NodeDeletionTaskManager for controllers to be deployed

    In case we remove controller from ha env we must redeploy others which
    is done in the manager. Check added to list of retrieved for deployment
    controllers in order to make sure they are in appropriate for the
    operation status. In opposite case we might faced bugs in particular
    situations, for example, when environment is reset and implicit
    deployment is started on nodes which still are undergoing the process and
    that leads to unexpected errorfull behaviour as such described in the bug

    Change-Id: I099660db1f8cd9d179f377799231635865286d3a
    Closes-Bug: #1432615

[Andrey Sledzinskiy (asledzinskiy)]

verified on
{

    "build_id": "288",
    "build_number": "288",
    "release_versions":

{

    "2015.1.0-7.0":

{

    "VERSION":

{

    "build_id": "288",
    "build_number": "288",
    "api": "1.0",
    "fuel-library_sha": "121016a09b0e889994118aa3ea42fa67eabb8f25",
    "nailgun_sha": "93477f9b42c5a5e0506248659f40bebc9ac23943",
    "feature_groups":

            [
                "mirantis"
            ],
            "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
            "openstack_version": "2015.1.0-7.0",
            "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d",
            "production": "docker",
            "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac",
            "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95",
            "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045",
            "release": "7.0",
            "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"
        }
    }

},
"auth_required": true,
"api": "1.0",
"fuel-library_sha": "121016a09b0e889994118aa3ea42fa67eabb8f25",
"nailgun_sha": "93477f9b42c5a5e0506248659f40bebc9ac23943",
"feature_groups":

    [
        "mirantis"
    ],
    "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd",
    "openstack_version": "2015.1.0-7.0",
    "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d",
    "production": "docker",
    "python-fuelclient_sha": "1ce8ecd8beb640f2f62f73435f4e18d1469979ac",
    "astute_sha": "a717657232721a7fafc67ff5e1c696c9dbeb0b95",
    "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045",
    "release": "7.0",
    "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"

}