Several deployment tasks stuck in running state

[Vladimir]

Several deplyment tasks never switch from running to ready state after successful deployment.
Steps to reprocude:
1. Create new env.
2. Add one controller node and deploy changes.
3. Check deployment tasks status after successful deployment.

Expecte4d result:
All tasks switch from running to ready state. time_end fiel contains adequate value.

Actual result:
Several tasks stuck in runnig state. time_end contains "None".

[root@nailgun ~]# fuel deployment-tasks --task-id 38 --status running
deployment_graph_task_name | node_id | status | time_start | time_end
-------------------------------|---------|---------|----------------------------|---------
memcached | 1 | running | 2016-03-31T07:54:44.486212 | None
primary-keystone | 1 | running | 2016-03-31T07:55:14.767550 | None
enable_quorum | 1 | running | 2016-03-31T08:15:55.071419 | None
dump_rabbitmq_definitions | 1 | running | 2016-03-31T08:16:07.594486 | None
nova-keystone | 1 | running | 2016-03-31T07:58:35.171956 | None
cinder-keystone | 1 | running | 2016-03-31T08:03:52.589336 | None
primary-ceph-mon | 1 | running | 2016-03-31T08:06:45.174647 | None
reserved_ports | 1 | running | 2016-03-31T07:46:16.241846 | None
fuel_pkgs | 1 | running | 2016-03-31T07:46:44.452711 | None
swift-keystone | 1 | running | 2016-03-31T08:10:36.415156 | None
netconfig | 1 | running | 2016-03-31T07:47:18.772263 | None
virtual_ips | 1 | running | 2016-03-31T07:49:38.360901 | None
openstack-haproxy-radosgw | 1 | running | 2016-03-31T07:50:24.039013 | None
radosgw-keystone | 1 | running | 2016-03-31T08:13:46.691748 | None
openstack-haproxy-aodh | 1 | running | 2016-03-31T07:50:41.224444 | None
openstack-haproxy-horizon | 1 | running | 2016-03-31T07:50:54.102146 | None
api-proxy | 1 | running | 2016-03-31T08:13:54.799871 | None
create-cinder-types | 1 | running | 2016-03-31T08:14:48.037582 | None
disable_keystone_service_token | 1 | running | 2016-03-31T08:16:13.047770 | None
openstack-network-networks | 1 | running | 2016-03-31T08:02:52.184684 | None
rsync_core_puppet | 1 | running | 2016-03-31T07:46:00.457990 | None
primary-swift-proxy | 1 | running | 2016-03-31T08:12:57.268487 | None
openstack-haproxy-heat | 1 | running | 2016-03-31T07:51:15.090813 | None

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Roman Prykhodchenko (romcheg)]

Could you please attach a diagnostic snaphot?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/300165

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/300155
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=cbc50dbebd1edb527b7041df8730de3c5c526c39
Submitter: Jenkins
Branch: master

commit cbc50dbebd1edb527b7041df8730de3c5c526c39
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Thu Mar 31 21:49:09 2016 +0300

    Report task status if it was changed

    For some reason we block send status report because of cache.
    It should protectd Nailgun from overload, but we should respect
    task status change in case of cache invalidation.

    Change-Id: If90151c846c239732097c53e358c97e3a79abf88
    Closes-Bug: #1564295

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/300165
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=0db640f05bfb43ec2a6968c31965dbbc128d9743
Submitter: Jenkins
Branch: master

commit 0db640f05bfb43ec2a6968c31965dbbc128d9743
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Thu Mar 31 22:13:11 2016 +0300

    Add warning if no task in history was found

    For some reason Receiverd has got report from Astute,
    but do not process them. This warn will give us full
    info about update task process in deployemnt history

    Change-Id: I475f45d43beb11cca61e65e0669ba821bdaea807
    Related-Bug: #1564295

[Vladimir (vushakov)]

Verified on:
    Fuel 9.0 ISO #157

[Aleksei Stepanov (penguinolog)]

https://custom-ci.infra.mirantis.net/job/9.0.custom.system_test/40/console
https://custom-ci.infra.mirantis.net/job/9.0.custom.system_test/41/console

Patch for fast check:
https://review.openstack.org/#/c/301589/3

[Aleksei Stepanov (penguinolog)]

Several tasks still in pending state.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/302898

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/302898
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=5415b86ac18750ce411898e46e0478b125e10079
Submitter: Jenkins
Branch: master

commit 5415b86ac18750ce411898e46e0478b125e10079
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Thu Apr 7 18:32:31 2016 +0300

    Always report running status for task

    Sometime we run task so fast, so we get only pending
    and succeed/failed status report without running.

    Such behavior breaks State Machine flow for tasks history
    in Nailgun. With this change we will send running status
    for task in the moment of actual run.

    Change-Id: I46d86e31b5e35a54d1c33e41ea6fbdbc0d74c66b
    Closes-Bug: #1564295

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/304051

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/304051
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=3c5275bf7613e507e8db24f0f364f24cf59ce60d
Submitter: Jenkins
Branch: stable/mitaka

commit 3c5275bf7613e507e8db24f0f364f24cf59ce60d
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Thu Apr 7 18:32:31 2016 +0300

    Always report running status for task

    Sometime we run task so fast, so we get only pending
    and succeed/failed status report without running.

    Such behavior breaks State Machine flow for tasks history
    in Nailgun. With this change we will send running status
    for task in the moment of actual run.

    Change-Id: I46d86e31b5e35a54d1c33e41ea6fbdbc0d74c66b
    Closes-Bug: #1564295
    (cherry picked from commit 5415b86ac18750ce411898e46e0478b125e10079)

[Aleksei Stepanov (penguinolog)]

fuel-9.0-203-2016-04-15_02-00-00.iso - FAIL.
https://review.openstack.org/#/c/301589/6 - checker

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/307874

[Aleksey Kasatkin (alekseyk-ru)]

I don't see why is it critical. Just deployment history is affected.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/310242

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/307874
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=f5dd8c43ed7c47489f89e9e2f5f070b8713279d6
Submitter: Jenkins
Branch: master

commit f5dd8c43ed7c47489f89e9e2f5f070b8713279d6
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Tue Apr 19 18:22:37 2016 +0300

    Flush DB must be run after all changes for nodes and histories

    Also add tests for state transition

    Change-Id: I4b3f63de0c96aebb3ca09e57a62b06895694b09d
    Related-Bug: #1564295

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/310433

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310242
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=1e913924a18638a620ba42ff0eb890cccf959f46
Submitter: Jenkins
Branch: master

commit 1e913924a18638a620ba42ff0eb890cccf959f46
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Tue Apr 26 18:54:50 2016 +0300

    Do not report 'running' for skipped tasks

    Without this change Astute will send running
    status for task which not run because skipped

    Change-Id: Icde12c45b2816aaa5facc26764d7dcb4792f1702
    Closes-Bug: #1564295

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/310481

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310481
Committed: https://git.openstack.org/cgit/openstack/fuel-astute/commit/?id=0027f0772535df87ff6fd4ce1b671582f0be4a67
Submitter: Jenkins
Branch: stable/mitaka

commit 0027f0772535df87ff6fd4ce1b671582f0be4a67
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Tue Apr 26 18:54:50 2016 +0300

    Do not report 'running' for skipped tasks

    Without this change Astute will send running
    status for task which not run because skipped

    Change-Id: Icde12c45b2816aaa5facc26764d7dcb4792f1702
    Closes-Bug: #1564295
    (cherry picked from commit 1e913924a18638a620ba42ff0eb890cccf959f46)

[Aleksei Stepanov (penguinolog)]

Custom PASS on both ISO

[Dmitriy Kruglov (dkruglov)]

Occurrences on the latest MOS 9.0 swarm (ISO build 278):
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.bvt_ubuntu_bootstrap/98/testReport/(root)/delete_node_on_ready_ubuntu_bootstrap/delete_node_on_ready_ubuntu_bootstrap/
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.bvt_ubuntu_bootstrap/98/testReport/(root)/deploy_with_two_ubuntu_bootstraps/deploy_with_two_ubuntu_bootstraps/
https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.jumbo_frames/99/testReport/(root)/jumbo_frames_neutron_diff_net_vxlan/jumbo_frames_neutron_diff_net_vxlan/

Also this bug directly affects LCM feature (implementation autotests for verification of idempotency and ensurability).

Raising to critical.

[Dmitriy Kruglov (dkruglov)]

Here are the occurrences on 10.0 swarm:
https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.bvt_ubuntu_bootstrap/27/testReport/junit/(root)/delete_node_on_ready_ubuntu_bootstrap/delete_node_on_ready_ubuntu_bootstrap/
https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.services_reconfiguration_thread_2/28/testReport/junit/(root)/basic_env_for_reconfiguration/basic_env_for_reconfiguration/
https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.thread_non_func_1/27/testReport/junit/(root)/deploy_ha_one_controller_neutron_master_node_fail/deploy_ha_one_controller_neutron_master_node_fail/
https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.thread_3/27/testReport/junit/(root)/ceph_ha/ceph_ha/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/310433
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=67d261a9eada140d572a6dedb64f28fbb6bbd4dd
Submitter: Jenkins
Branch: stable/mitaka

commit 67d261a9eada140d572a6dedb64f28fbb6bbd4dd
Author: Vladimir Sharshov (warpc) <email address hidden>
Date: Tue Apr 19 18:22:37 2016 +0300

    Flush DB must be run after all changes for nodes and histories

    Also add tests for state transition

    Change-Id: I4b3f63de0c96aebb3ca09e57a62b06895694b09d
    Related-Bug: #1564295
    (cherry picked from commit f5dd8c43ed7c47489f89e9e2f5f070b8713279d6)

[Vladimir Sharshov (vsharshov)]

We have several different bug here:

1. https://bugs.launchpad.net/fuel/+bug/1578218

- https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.thread_3/27/testReport/junit/(root)/ceph_ha/ceph_ha/
- https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.thread_non_func_1/27/testReport/junit/(root)/deploy_ha_one_controller_neutron_master_node_fail/deploy_ha_one_controller_neutron_master_node_fail/
- https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.services_reconfiguration_thread_2/28/testReport/junit/(root)/basic_env_for_reconfiguration/basic_env_for_reconfiguration/

2. Another problem https://bugs.launchpad.net/fuel/+bug/1578257

- https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.bvt_ubuntu_bootstrap/27/testReport/junit/(root)/delete_node_on_ready_ubuntu_bootstrap/delete_node_on_ready_ubuntu_bootstrap/ do not update status for Nailgun task (not deployment history)

So i move this bug to fix committed state. Please check bugs above to find similar problem.

[Bogdan Dobrelya (bogdando)]

CI is still blocked by this bug

[Bogdan Dobrelya (bogdando)]

Example failures in stable/mitaka, see the recent recheck runs:
https://review.openstack.org/#/c/313674 https://review.openstack.org/#/c/313273/

[Vladimir Sharshov (vsharshov)]

Guys, please check this bugs: https://bugs.launchpad.net/fuel/+bug/1578218 and https://bugs.launchpad.net/fuel/+bug/1578257 first before reopen this.

[ElenaRossokhina (esolomina)]

Verified: all tasks have ready or skipped status, no running and pending tasks after deploy ends
iso#305 http://paste.openstack.org/show/496749/

[Dmitry Belyaninov (dbelyaninov)]

Verified on Newton, iso #1332
There is no running tasks after deployment

[root@nailgun ~]# fuel deployment-tasks --task-id 5 --status running
task_name | node_id | status | time_start | time_end
----------+---------+--------+------------+---------