stop deployment doesn't work if started before start provisioning

[Artem Hrechanychenko]

stop deployment doesn't work if started before provisioning begin

Step's to reproduce:
1) Create any cluster
2) Verify networks
3) Start deployment
4) Stop deployment before provisioning begins

Expected results:
Deployment stopped succesfully.

Actual results:
Deployment proccess occurs in a regular mode

Astute logs:

2015-09-23 08:36:55 INFO [646] Processing RPC call 'image_provision'
2015-09-23 08:36:48 INFO [658] Processing RPC call 'stop_deploy_task'
2015-09-23 08:36:46 INFO [642] Processing RPC call 'stop_deploy_task'
2015-09-23 08:36:45 INFO [646] Processing RPC call 'stop_deploy_task'
2015-09-23 08:36:44 INFO [654] Processing RPC call 'stop_deploy_task'
2015-09-23 08:36:41 INFO [636] Processing RPC call 'stop_deploy_task'
2015-09-23 08:36:41 INFO [650] Processing RPC call 'stop_deploy_task'
2015-09-23 08:36:41 INFO [634] Processing RPC call 'stop_deploy_task'

Status bar show "Current task: Stopping..."

Provisioning was successfully done and then start deployment OpenStack proccess

[root@nailgun ~]# fuel task
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
id | status | name | cluster | progress | uuid
----|---------|------------------------------------|---------|----------|-------------------------------------
48 | ready | dump | None | 100 | 292ed9cd-25d3-4fac-bf8a-c5480e398d52
111 | ready | check_repo_availability_with_setup | 7 | 100 | 2d0ea5b1-4859-45ec-848c-729d2d72391a
108 | ready | verify_networks | 7 | 100 | 0ea8fcb9-3e9b-470b-ac53-d3cc4db34bbb
112 | ready | deploy | 7 | 45 | 70b0b68a-5d9f-48fb-849c-6a05f14bc6dc
109 | ready | check_dhcp | 7 | 100 | 3dfc5bd8-336a-414f-8305-c5db54039251
110 | ready | check_repo_availability | 7 | 100 | 731360dc-9019-4718-a342-e44b6de3ab36
117 | running | stop_deployment | 7 | 0 | c9a7f8c6-c5e7-49c1-b25b-04157c76c5bc
113 | ready | check_networks | 7 | 100 | d603ed6f-4703-418d-a7f7-ccf121b1831d
118 | running | deployment | 7 | 0 | 76e9654b-1d71-4698-8818-411beaf84965
116 | running | provision | 7 | 17 | f8bb3242-c453-431b-a16f-94e041f2e465
[root@nailgun ~]# fuel task
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
id | status | name | cluster | progress | uuid
----|---------|------------------------------------|---------|----------|-------------------------------------
119 | ready | dump | None | 100 | c156eaf2-54eb-45e1-8d9f-fa6ee4eb1c76
111 | ready | check_repo_availability_with_setup | 7 | 100 | 2d0ea5b1-4859-45ec-848c-729d2d72391a
108 | ready | verify_networks | 7 | 100 | 0ea8fcb9-3e9b-470b-ac53-d3cc4db34bbb
109 | ready | check_dhcp | 7 | 100 | 3dfc5bd8-336a-414f-8305-c5db54039251
110 | ready | check_repo_availability | 7 | 100 | 731360dc-9019-4718-a342-e44b6de3ab36
117 | running | stop_deployment | 7 | 0 | c9a7f8c6-c5e7-49c1-b25b-04157c76c5bc
112 | ready | deploy | 7 | 68 | 70b0b68a-5d9f-48fb-849c-6a05f14bc6dc
113 | ready | check_networks | 7 | 100 | d603ed6f-4703-418d-a7f7-ccf121b1831d
116 | ready | provision | 7 | 100 | f8bb3242-c453-431b-a16f-94e041f2e465
118 | running | deployment | 7 | 22 | 76e9654b-1d71-4698-8818-411beaf84965

2015-09-21 12:42:31 INFO [658] Casting message to Nailgun: {"method"=>"provision_resp", "args"=>{"task_uuid"=>"eebcd269-74e4-4b44-a04c-f6b36ba9f229", "nodes"=>[{"uid"=>"29", "progress"=>100, "status"=>"provisioned"}, {"uid"=>"30", "progress"=>100, "status"=>"provisioned"}, {"uid"=>"31", "progress"=>100, "status"=>"provisioned"}], "status"=>"ready", "progress"=>100}}

2015-09-21 12:42:31 INFO [658] Processing RPC call 'granular_deploy'

2015-09-21 12:44:53 INFO [658] Casting message to Nailgun: {"method"=>"deploy_resp", "args"=>{"task_uuid"=>"300c7552-a8a4-41a5-af8e-1f376a33cfe7", "nodes"=>[{"uid"=>"29", "progress"=>0, "status"=>"deploying", "role"=>"primary-controller", "task"=>{"priority"=>700, "type"=>"puppet", "uids"=>["29"], "parameters"=>{"puppet_modules"=>"/etc/puppet/modules", "puppet_manifest"=>"/etc/puppet/modules/osnailyfacter/modular/netconfig/netconfig.pp", "timeout"=>3600, "cwd"=>"/"}}}]}}

{"build_id": "298", "build_number": "298", "release_versions": {"2015.1.0-7.0": {"VERSION": {"build_id": "298", "build_number": "298", "api": "1.0", "fuel-library_sha": "0623b4daad438ceeb5dc41b10cdd3011795fff7e", "nailgun_sha": "d590b26dbb09785b8a8b3651b0ef69746fcf9991", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}}}, "auth_required": true, "api": "1.0", "fuel-library_sha": "0623b4daad438ceeb5dc41b10cdd3011795fff7e", "nailgun_sha": "d590b26dbb09785b8a8b3651b0ef69746fcf9991", "feature_groups": ["mirantis"], "fuel-nailgun-agent_sha": "d7027952870a35db8dc52f185bb1158cdd3d1ebd", "openstack_version": "2015.1.0-7.0", "fuel-agent_sha": "082a47bf014002e515001be05f99040437281a2d", "production": "docker", "python-fuelclient_sha": "486bde57cda1badb68f915f66c61b544108606f3", "astute_sha": "6c5b73f93e24cc781c809db9159927655ced5012", "fuel-ostf_sha": "1f08e6e71021179b9881a824d9c999957fcc7045", "release": "7.0", "fuelmain_sha": "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"}

[Alexander Gordeev (a-gordeev)]

Confirmed. Since it is just high, it will be moved to 8.0, for 7.0 it stays as WONT FIX as only critical one could be included into 7.0 after HCF deadline.

Otherwise, don't hesitate to argue why it's critical for 7.0

[Alexander Kislitsky (akislitsky)]

The 'stop_deploy_task' was passed to astute before provisioning task. Thus all astute workers just consume task and did nothing.
For fix this issue we should send stop tasks from nailgun only after provision or deploy tasks has been sent.

[Alexander Kislitsky (akislitsky)]

Creation of task in the Nailgun DB and sending message to Astute are independent. We can have task in DB, but this task is not sent into Astute. Also we have no information about sending task into Astute. Thus in the current implementation of task processing this issue can't be fixed without introducing of some ugly and sophisticated timeouts into StopDeploymentTask.

This bug should be fixed as part of feature 'Changing transactions workflow in the Nailgun'. As part of the feature we should refactor tasks processing: only messages generation should be used, DB commits should be removed from tasks processing, all messages to Astute should be sent at the end of TaskManager method execution. Thus we will control task processing in the transactional manner and in case of race between provisioning/deployment and stop deployment tasks the stop deployment will raise an error - 'Nothing to stop': https://github.com/stackforge/fuel-web/blob/c77efaaf1d8f29eff79d25f3f7d6bd3a496ae263/nailgun/nailgun/task/manager.py#L652-L657

[Vladimir Sharshov (vsharshov)]

After discussion with Alexander Kislitsky and Evgeniy L we suggest such solution.

- add new field for task in DB 'received', which default is false;
- add new behavior in Astute which must report received status about taken task;
- change stop_deployment task in Nailgun to check received field before send stop_deployment task.

Potential problem and solutions:

1. In case of stop_deployment task we will get several received status instead of one. It should not be a problem, but we can exclude such behavior for service call or send it only if it match the task to stop from one of the worker.
2. If received status was not set, we have 2 variant to react: wait for some time after it check again or fail stop_deployment task.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/229921

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/229953

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/230747

[OpenStack Infra (hudson-openstack)]

Change abandoned by Alexander Kislitsky (<email address hidden>) on branch: master
Review: https://review.openstack.org/230747
Reason: Fix proposed in https://review.openstack.org/#/c/229953/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/239979

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/239979
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=8e8f21fed9787781b02f46a72e82b04ee5523241
Submitter: Jenkins
Branch: master

commit 8e8f21fed9787781b02f46a72e82b04ee5523241
Author: Alexander Kislitsky <email address hidden>
Date: Wed Oct 28 19:03:36 2015 +0300

    Task status 'pending' handled in tests

    Status 'pending' introduced for proper tasks as intermediate status
    for tasks ready for sending or already sent to astute, but not
    acknoledged by astute. After task is acknoledged by astute it
    changes status to 'running'.

    Change-Id: Idedb061b7b5c4dca4a0ca7adcaa570cecbb691af
    Partial-Bug: #1498827

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229921
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=3067b1f5d53ad1e2bccec685b53d0f6c9c0de100
Submitter: Jenkins
Branch: master

commit 3067b1f5d53ad1e2bccec685b53d0f6c9c0de100
Author: Alexander Kislitsky <email address hidden>
Date: Thu Oct 1 17:36:59 2015 +0300

    Sending confirmation to Nailgun on task dispatching

    We need to know is task handled by orchestrator in the Nailgun
    or not. For instance we shouldn't send stop task, if provsion or
    deploy tasks not handled by the orchestrator.

    Change-Id: Ib054517696dc4e53487557b09b75ebfcb1255ecb
    Partial-Bug: #1498827

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229953
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=fa46c4dca938acc2920f6937480fc9b4b44deca7
Submitter: Jenkins
Branch: master

commit fa46c4dca938acc2920f6937480fc9b4b44deca7
Author: Alexander Kislitsky <email address hidden>
Date: Wed Sep 30 14:14:26 2015 +0300

    Receiving of task processing confirmation from astute handled

    We should know is task handled by orchestrator or not.
    For instance, we should send stop_deployment task only if
    provision or deploy tasks are handled by the orchestrator.
    Task status 'pending' added into Task DB model and
    handled in stop deployment, provisioning, deployment and
    apply cluster changes task managers.

    Nailgun Task object update function changed for bubble 'running'
    status to parent task.

    Locking of all cluster tasks calls removed for deadlocks
    exclusion.

    Consts used instead hardcoded tasks statuses in part of tests.

    Co-Authored-By: Alexandra Morozova <email address hidden>
    Depends-On: Ib054517696dc4e53487557b09b75ebfcb1255ecb
    Depends-On: Idedb061b7b5c4dca4a0ca7adcaa570cecbb691af
    Change-Id: I15ebeb85226c832923f9476bb91fa19c0ff87a4f
    Closes-Bug: #1498827

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/240292
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=a25263937d2ef9acba55e71f883205ee8d026e1e
Submitter: Jenkins
Branch: master

commit a25263937d2ef9acba55e71f883205ee8d026e1e
Author: Alexandra Morozova <email address hidden>
Date: Fri Oct 30 10:26:22 2015 +0100

    Forbid stopping of task with 'pending' status

    Related-Bug: #1498827

    Change-Id: Id150d7bace2af09e6615afff9cb1bc4cabd84538

[Artem Hrechanychenko (agrechanichenko)]

Verified on build ISO #127