"ready" status while post_deploy tasks still running

Bug #1439776 reported by Vyacheslav Struk on 2015-04-02
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Critical
Vladimir Sharshov
Mitaka
Critical
Vladimir Sharshov

Bug Description

Some plugins can run post_deploy task for a significant time, but web UI does not reflects this fact.
State of all the nodes is "ready", but post_deploy tasks are still running.

Such behaviour makes user think that everything is up and running when it's not.

Proposed solution:

Track progress of post_deploy tasks and set "ready" state when all the tasks are really finished.

description: updated
Changed in fuel:
milestone: none → 6.1
assignee: nobody → Fuel Python Team (fuel-python)
importance: Undecided → High
Dmitry Pyzhov (dpyzhov) on 2015-04-08
Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Vladimir Sharshov (vsharshov)
status: New → Confirmed
Dmitry Pyzhov (dpyzhov) on 2015-04-08
tags: added: module-tasks
Dmitry Pyzhov (dpyzhov) on 2015-04-08
Changed in fuel:
assignee: Vladimir Sharshov (vsharshov) → Dima Shulyak (dshulyak)
Vladimir Sharshov (vsharshov) wrote :

At now moment we set 'ready' for nodes after deploy stage. Post deploy stage works with ready nodes (and use ready status for own proposes). At same time cluster status is deploying.

Dima Shulyak (dshulyak) wrote :

What do you think about changing status of the node to running,
when post deployment task will start and change it back after it will finish?

I am afraid that it may be even more confusing that some node will constantly change its state during post deployment,
but atleast it is some feedback on UI

Dmitry Pyzhov (dpyzhov) on 2015-04-24
Changed in fuel:
assignee: Dima Shulyak (dshulyak) → Fuel Python Team (fuel-python)
Vladimir Sharshov (vsharshov) wrote :

Some tasks for at least for 6.0 can run or not based on node ready status. I think this is task for 7.0 where we can remake report system based on tasks (instead of progress bar with 0..100 we can show <done tasks>/<all tasks> ) with name of executing task.

Like:

     4/17 (puppet controller)

I vote to move it to 7.0.

Nikolay Markov (nmarkov) wrote :

+1 to vsharshov. Also, I think we need once again discuss connection of status in UI/CLI with list of pending changes on environment. A lot of things like this happen because we're checking status based on some complex business logic instead of just looking at one place.

Dmitry Pyzhov (dpyzhov) on 2015-04-29
tags: added: release-notes
Changed in fuel:
milestone: 6.1 → 7.0
no longer affects: fuel/6.1.x
Dmitry Pyzhov (dpyzhov) on 2015-04-29
tags: added: feature
Aleksandr Didenko (adidenko) wrote :

Forwarded to Vova to link with BP.

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Vladimir Sharshov (vsharshov)
tags: added: covered-by-bp
Changed in fuel:
assignee: Vladimir Sharshov (vsharshov) → Fuel Python Team (fuel-python)
Changed in fuel:
milestone: 7.0 → 8.0
tags: added: qa-agree-8.0
tags: removed: feature
Dmitry Pyzhov (dpyzhov) on 2015-10-22
tags: added: area-python
Changed in fuel:
milestone: 8.0 → 9.0

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

tags: added: need-info
tags: removed: need-info
Vladimir Sharshov (vsharshov) wrote :

This bug was solved by moving to new deployment engine: task based deployment.

This feature was introduced in 8.0 as experimental and now use as defualt: https://github.com/openstack/fuel-specs/blob/master/specs/8.0/task-based-deployment-mvp.rst

Changed in fuel:
assignee: Fuel Python Team (fuel-python) → Vladimir Sharshov (vsharshov)
status: Confirmed → Fix Committed

Reviewed: https://review.openstack.org/304612
Committed: https://git.openstack.org/cgit/openstack/fuel-docs/commit/?id=e9d1be7df3a46bfa5d40229dcdf73c64864f1be6
Submitter: Jenkins
Branch: master

commit e9d1be7df3a46bfa5d40229dcdf73c64864f1be6
Author: Evgeny Konstantinov <email address hidden>
Date: Tue Apr 12 16:15:08 2016 +0300

    Add Fuel Mitaka known issues to relnotes
    Related-Bug: #1439776
    Related-Bug: #1450100
    Related-Bug: #1460169
    Related-Bug: #1490597
    Related-Bug: #1526544
    Related-Bug: #1556854
    Related-Bug: #1446704

    Change-Id: I3df16c163d82af7d0db8a64643b915909cabd8f1

Alexey Stepanov (astepanov-m) wrote :
Download full text (3.9 KiB)

https://custom-ci.infra.mirantis.net/job/9.0.custom.system_test/132/console

Part of output:
Node: 4
   Task name: openstack-network-end
    Status: running
    Start: 2016-04-15T12:49:47.529120
    End: None

   Task name: vmware-vcenter
    Status: running
    Start: 2016-04-15T12:49:55.707880
    End: None

   Task name: murano-cfapi
    Status: running
    Start: 2016-04-15T12:49:56.289863
    End: None

   Task name: plugins_rsync
    Status: running
    Start: 2016-04-15T12:14:25.017842
    End: None

   Task name: murano-rabbitmq
    Status: running
    Start: 2016-04-15T12:26:17.330997
    End: None

   Task name: aodh
    Status: running
    Start: 2016-04-15T12:30:10.389962
    End: None

   Task name: upload_nodes_info
    Status: running
    Start: 2016-04-15T12:51:59.609660
    End: None

   Task name: ceilometer-radosgw-user
    Status: running
    Start: 2016-04-15T12:52:15.266825
    End: None

   Task name: ironic-api
    Status: running
    Start: 2016-04-15T12:31:06.480011
    End: None

   Task name: openstack-haproxy-ironic
    Status: running
    Start: 2016-04-15T12:20:18.511069
    End: None

   Task name: openstack-network-start
    Status: running
    Start: 2016-04-15T12:39:22.907771
    End: None

   Task name: murano
    Status: running
    Start: 2016-04-15T12:49:47.661743
    End: None

   Task name: swift-storage
    Status: running
    Start: 2016-04-15T12:49:47.918049
    End: None

   Task name: sahara
    Status: running
    Start: 2016-04-15T12:49:48.055801
    End: None

   Task name: ironic-compute
    Status: running
    Start: 2016-04-15T12:49:55.933839
    End: None

   Task name: swift-rebalance-cron
    Status: running
    Start: 2016-04-15T12:49:56.124959
    End: None

   Task name: swift-proxy
    Status: running
    Start: 2016-04-15T12:42:33.317464
    End: None

   Task name: plugins_setup_repositories
    Status: running
    Start: 2016-04-15T12:14:25.589685
    End: None

   Task name: copy_haproxy_keys
    Status: running
    Start: 2016-04-15T12:14:38.697884
    End: None

   Task name: ceilometer-controller
    Status: running
    Start: 2016-04-15T12:30:11.457937
    End: None

  Node: 5
   Task name: openstack-network-end
    Status: running
    Start: 2016-04-15T12:46:06.828764
    End: None

   Task name: openstack-network-start
    Status: running
    Start: 2016-04-15T12:42:30.963689
    End: None

   Task name: upload_nodes_info
    Status: running
    Start: 2016-04-15T12:50:08.704074
    End: None

   Task name: ceilometer-compute
    Status: running
    Start: 2016-04-15T12:44:16.146016
    End: None

   Task name: plugins_rsync
    Status: running
    Start: 2016-04-15T12:14:23.565426
    End: None

   Task name: plugins_setup_repositories
    Status: running
    Start: 2016-04-15T12:14:37.193611
    End: None

   Task name: copy_haproxy_keys
    Status: running
    Start: 2016-04-15T12:14:37.825295
    End: None

  Node: 6
   Task name: openstack-network-end
    Status: running
    Start: 2016-04-15T12:46:10.625376
    End: None

   Task name: upload_nodes_info
   ...

Read more...

Changed in fuel:
status: Fix Committed → Confirmed
Alexey Stepanov (astepanov-m) wrote :

ISO 9.0.132

Changed in fuel:
importance: High → Critical
Changed in fuel:
milestone: 9.0 → 10.0
Vladimir Sharshov (vsharshov) wrote :

This is not the same problem. Please check this bug: https://bugs.launchpad.net/fuel/+bug/1564295 for details.

Changed in fuel:
status: Confirmed → Fix Committed
tags: added: on-verification

verified on 9.0-305 iso

Related fix proposed to branch: master
Change author: Evgeny Konstantinov <email address hidden>
Review: https://review.fuel-infra.org/22324

Reviewed: https://review.fuel-infra.org/22324
Submitter: Evgeny Konstantinov <email address hidden>
Branch: master

Commit: 12616554a46a3b751ced584026da92713d87c8ed
Author: Evgeny Konstantinov <email address hidden>
Date: Wed Jun 22 14:24:14 2016

Add Fuel resolved issues to 9.0

Change-Id: Id80cf3e87dda6aecd9bb845d75a65e23aeee2b3e
Related-Bug: #1439776
Related-Bug: #1539503

tags: added: release-notes-done
removed: release-notes
Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Related blueprints