Unable to continue deployment after it has been stopped

[Vitaly Kramskikh]

ISO: {"build_id": "2014-05-15_01-10-31", "mirantis": "yes", "build_number": "205", "ostf_sha": "69bdb9729cbf16d2c8ba927e946d64a8898909af", "nailgun_sha": "0c88805f28d658be82b305eaeafc5466b563569d", "production": "docker", "api": "1.0", "fuelmain_sha": "fc249cd1fe62d5303f531cf00d9e31ee2ab642d2", "astute_sha": "2b19f9cda281a4e53fb4a2e241d882e29ab9724b", "release": "5.0", "fuellib_sha": "1c7d598a65309f8bda1391360f0b94bfb5015180"}

Steps:
1. Create env (Centos, simple mode, 1 compute, 1 controller, 3 ceph)
2. Start deployment
3. Wait for provisioning of all nodes and 50% deployment of the controller node
4. Stop deployment
5. After deployment successfully stop, try to start it again

Expected result:
1. Deployment continues

Actual result:
1. Changes handler returns 400 error with "No changes to deploy" text
2. After a few mitues nodes' status becomes "discovered" without pending_addition flag

[Aleksey Kasatkin (alekseyk-ru)]

Reproduced on ISO #205. I see the same problem after STOP, then RESET - deployment cannot be started.

[Aleksey Kasatkin (alekseyk-ru)]

Deploy - Stop - Deploy(error) - Reset - Deploy(error)

[Aleksey Kasatkin (alekseyk-ru)]

Nodes' roles are not returned to pending roles. It seems to be the root cause.

[root@fuel ~]# \fuel node
id | status | name | cluster | ip | mac | roles | pending_roles | online
---|----------|------------------|---------|-------------|-------------------|------------|---------------|-------
4 | discover | Untitled (06:00) | 2 | 10.20.0.174 | 08:00:27:67:06:00 | controller | | True
9 | discover | Untitled (68:2f) | 2 | 10.20.0.190 | 08:00:27:68:68:2f | | cinder | True
5 | discover | Untitled (ac:10) | 2 | 10.20.0.143 | 08:00:27:58:ac:10 | compute | | True
6 | discover | Untitled (51:ba) | 2 | 10.20.0.195 | 08:00:27:34:51:ba | cinder | | True

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/93940

[Vladimir Sharshov (vsharshov)]

Reproduced on #211

[Aleksey Kasatkin (alekseyk-ru)]

Problem seems to be not on nailgun side. Message from orchestrator doesn't contain nodes IDs.

2014-05-15 12:56:29.037 INFO [7f2a786dc700] (receiver) RPC method stop_deployment_resp received: {"status": "ready", "progress": 100, "task_uuid": "0ed3948d-1409-4989-ba48-d02faea105a7"}

[Aleksey Kasatkin (alekseyk-ru)]

Nodes parameters were send to orchestrator on stop command and nodes were returned to bootstrap but there were no nodes in orchestrator's answer. Orchestrator's answer must contain list of nodes stopped.

To orchestrator:
2014-05-15 12:56:14.192 DEBUG [7f484dde5700] (__init__) RPC cast to orchestrator:
{
    "args": {
        "engine": {
            "url": "http://10.20.0.2:80/cobbler_api",
            "username": "cobbler",
            "password": "cobbler",
            "master_ip": "10.20.0.2"
        },
        "task_uuid": "0ed3948d-1409-4989-ba48-d02faea105a7",
        "stop_task_uuid": "020ba321-4f94-406f-af78-8a8e5a59b11a",
        "nodes": [
            {
                "uid": "4",
                "roles": [
                    "compute"
                ],
                "slave_name": "node-4"
            },
            {
                "uid": "3",
                "roles": [
                    "ceph-osd"
                ],
                "slave_name": "node-3"
            },
            {
                "uid": "5",
                "roles": [
                    "ceph-osd"
                ],
                "slave_name": "node-5"
            },
            {
                "uid": "2",
                "roles": [
                    "controller"
                ],
                "slave_name": "node-2"
            },
            {
                "uid": "1",
                "roles": [
                    "ceph-osd"
                ],
                "slave_name": "node-1"
            }
        ]
    },
    "respond_to": "stop_deployment_resp",
    "method": "stop_deploy_task",
    "api_version": "1.0"
}

From orchestrator:
2014-05-15 12:56:29.037 INFO [7f2a786dc700] (receiver) RPC method stop_deployment_resp received: {"status": "ready", "progress": 100, "task_uuid": "0ed3948d-1409-4989-ba48-d02faea105a7"}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/94163

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/94163
Committed: https://git.openstack.org/cgit/stackforge/fuel-astute/commit/?id=2a6655a0546aa608fb9fff7f4bbad2a65be462d2
Submitter: Jenkins
Branch: master

commit 2a6655a0546aa608fb9fff7f4bbad2a65be462d2
Author: Vladimir Sharshov <email address hidden>
Date: Mon May 19 14:29:46 2014 +0400

    Send data about erased nodes to Nailgun

    * fix code;
    * add test.

    Change-Id: I19eafbb0fb3d82536a1e19f1a46b7d5fddbef086
    Closes-Bug: #1319823

[Egor Kotko (ykotko)]

Verified on:
{"build_id": "2014-05-21_01-10-31", "mirantis": "yes", "build_number": "214", "ostf_sha": "353f918197ec53a00127fd28b9151f248a2a2d30", "nailgun_sha": "0b6e8eabaccad2aa29519561ce7cde9df9292964", "production": "docker", "api": "1.0", "fuelmain_sha": "910f262f85e94bef08e0e9b9d6230ad890bf139e", "astute_sha": "9a0d86918724c1153b5f70bdae008dea8572fd3e", "release": "5.0", "fuellib_sha": "3d92142a5643af82596f0450e39282550a45e5db"}

[Aleksey Kasatkin (alekseyk-ru)]

QA Team, please recheck this is fixed for 5.0.1 and 5.1.

[Anastasia Palkina (apalkina)]

Verified ob ISO #134
"build_id": "2014-07-17_00-31-14",
"mirantis": "yes",
"build_number": "134",
"ostf_sha": "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f",
"nailgun_sha": "1d08d6f80b6514085dd8c0af4d437ef5d37e2802",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "069686abb90f458f67cfcb4018cacc19971e4b4d",
"astute_sha": "9a74b788be9a7c5682f1c52a892df36e4766ce3f",
"release": "5.0.1",
"fuellib_sha": "2d1e1369c13bc9771e9473086cb064d257a21fc2"

[Dmitry Pyzhov (dpyzhov)]

Fix was provided in 5.0 release