reexecute_on: not starts task in post_deployment stage

[Nikita Koshikov]

Steps to reproduce:
1)Create plugin that inject 2 tasks in post_deployment stage, in this example - code below:
- id: set-compute-local-ephemeral
  type: puppet
  version: 2.0.0
  role: [compute]
  required_for: [post_deployment_end]
  requires: [ceph-compute]
  refresh_on: [nova_config, nova_paste_api_ini]
  parameters:
    puppet_manifest: "ephemeral-compute.pp"
    puppet_modules: "puppet/modules:/etc/puppet/modules"
    timeout: 120

- id: create-nova-aggregates
  type: puppet
  version: 2.0.0
  role: [primary-controller]
  required_for: [post_deployment_end]
  requires: [ceph_create_pools]
  reexecute_on: [deploy_changes]
  parameters:
    puppet_manifest: "create-aggregates.pp"
    puppet_modules: "puppet/modules:/etc/puppet/modules"
    timeout: 120
2)Create ENV and enable plugin
3)Deploy ENV
4)After deployment finish - add new node with "Compute" role
5)Deploy changes

Actual results - task 'create-nova-aggregates' is not executing on primary-controller.
Expected result - task should be executed.

This is MOS8 with role-based deployment.

[Ihor Kalnytskyi (ikalnytskyi)]

I've done some investigation and here's what I found:

* All nodes that are:

   * not going to be deployed
   * and not going to be deleted
   * and have 'ready' status

   will be selected for resetup.

* For resetup nodes, we're going to execute tasks marked with "reexecute_on: [deploy_changes]" parameter.

* The graph returns both pre- and post- and main deployment tasks.

* However, WE DO NOT send selected "reexecute_on" pre- and post- deployment tasks to Astute. We only mix main deployment tasks.

The buggy code:

    https://github.com/openstack/fuel-web/blob/26028971b493a4e47e18e8aa9c8b3a0fb91df8d8/nailgun/nailgun/task/manager.py#L401-L403

Fixing such code might have destructive effect for MU, since we do not guarantee that plugin node roles are ready to have "rexecuted_on" tasks provides by other plugins (if there're any with role: *).

[Nikita Koshikov (nkoshikov)]

Some more details:

If we remove node, without adding new ones - than tasks with reexecute_on: [deploy_changes] are executed, even for post_deployment_stage.

Steps to reproduce:
1) On deployed environment, remove compute.
2) Run deploy-changes

Actual results - task create-nova-aggregates is executed on controllers
Expected result - is the same

[Ihor Kalnytskyi (ikalnytskyi)]

The another interesting fact is, that we run "reexecut_on" pre-/post- tasks in downscale case: when we remove something and do not add anything.

So maybe fixing it in MU is not so dangerous as it seemed before.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/313076

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/313076
Committed: https://git.openstack.org/cgit/openstack/fuel-web/commit/?id=d35f0112d955e31e5ececdea17798b5be6ead435
Submitter: Jenkins
Branch: stable/8.0

commit d35f0112d955e31e5ececdea17798b5be6ead435
Author: Igor Kalnitsky <email address hidden>
Date: Thu May 5 20:09:29 2016 +0300

    Fix 'reexecute_on' for pre-/post- deployment tasks

    So far 'reexecute_on' for pre- and post- deployment tasks has been taken
    into account only in scale-down case (when there's nothing to deploy,
    only to remove). That cause a lot of trouble when it's impossible to
    plugin developers to trigger some action in scale-up case.

    Since now it's fixed, and 'reexecute_on' works for pre- and post-
    deployment tasks in all cases.

    This commit won't be merged into master and stable/9.0, since the whole
    system has been rewritten during LCM story and all tasks are intended to
    be reexecuted by default unless their conditions say "No".

    Change-Id: Ib65585c8b4cff18444192f377545fed82b5144da
    Closes-Bug: #1569920

[TatyanaGladysheva (tgladysheva)]

Could you please provide ephemeral-compute.pp and create-aggregates.pp puppet files for tasks 'set-compute-local-ephemeral', 'create-nova-aggregates'? It's not clear where I should take these files. Without it deployment fails.

[TatyanaGladysheva (tgladysheva)]

Verified on MOS 8.0 + MU2 updates.

Steps to verify was taken from bug description. *.pp files for tasks 'set-compute-local-ephemeral', 'create-nova-aggregates' were created with any puppet manifest content because it's not important.

Actual results:
According to /var/log/docker-logs/astute/astute.log, task "create-nova-aggregates" was executed on primary-controller during redeployment env after adding new compute node.