Detailed bug description:
After the upgrade of Fuel master node from 8 to 9.1 with the existing 8.0 cluster and the further scaling this cluster fails with error in the puppet's log:
Could not run: Could not find file /etc/puppet/modules/osnailyfacter/modular/hiera/hiera.pp
This happens because of
The root cause is task "rsync_core_puppet" doesn't execute and manifests are not delivered to the deploying node
Was found that granular_deploy modifies graph in case of some nodes are affected by deployment (e.g. in case of scaling a cluster):
2016-08-16 09:00:31.242 DEBUG [7f0057829880] (manager) There are nodes to deploy: node-9.test.domain.local
2016-08-16 09:00:31.285 DEBUG [7f0057829880] (manager) There are nodes affected by deployment: node-7.test.domain.local node-4.test.domain.local node-5.test.domain.local node-8.test.domain.local node-
2.test.domain.local node-6.test.domain.local node-3.test.domain.local node-1.test.domain.local
and this modification of the graph makes all non-reexecutable tasks as `skipped`:
...
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task rsync_core_puppet will be skipped.
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task clear_nodes_info will be skipped.
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task generate_keys will be skipped.
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task copy_keys will be skipped.
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task generate_haproxy_keys will be skipped.
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task copy_haproxy_keys will be skipped.
2016-08-16 14:59:03.298 DEBUG [7f0057829880] (orchestrator_graph) Task sync_time will be skipped.
...
after that the graph is modified to handle only affected nodes and is incorrect for nodes that have to be deployed.
Steps to reproduce:
1. Deploy 8.0 Fuel node
2. Create cluster with 3 cntrl + 3 compute+ceph-osd
3. Upgrade Fuel master node from 8.0 to 9.1 (http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-install-guide/upgrade/upgrade-fuel.html)
4. Add node with role "compute" to created cluster
5. Deploy changes
Expected result:
Deployment succeeds
Actual result:
Deployment fails as rsync_core_puppet task does not run on the newly added node.
This happens due to the fact that pre_|post_deployment stages do not contain any tasks. Here is a excerpt of debugged print from task.py for granular_deploy:
'post_deployment': [],
'pre_deployment': []
What's also important that the deployment succeeds if we provision the node first and deploy it then.
Workaround:
Some modifications can be done to have two graphs but it leads to run whole deployment graph on affected nodes too:
diff --git a/nailgun/nailgun/task/task.py b/nailgun/nailgun/task/task.py
index b5f3b6a..e985652 100644
--- a/nailgun/nailgun/task/task.py
+++ b/nailgun/nailgun/task/task.py
@@ -330,9 +330,10 @@ class DeploymentTask(BaseDeploymentTask):
cls._save_deployment_info(transaction, serialized_cluster)
if affected_nodes:
- graph.reexecutable_tasks(events)
+ reexec_graph = graph.copy()
+ reexec_graph.reexecutable_tasks(events)
serialized_cluster.extend(deployment_serializers.serialize(
- graph, transaction.cluster, affected_nodes
+ reexec_graph, transaction.cluster, affected_nodes
))
nodes = nodes + affected_nodes
pre_deployment = stages.pre_deployment_serialize(
Octane's versions:
8.0.0-1.mos1192
9.0.0-1.mos1208
ISO's versions:
8.0 - http://paste.openstack.org/show/538992/
9.1 - http://paste.openstack.org/show/542518/
It seems that some regression of functionality in 9.0 was done for the granular_deployment method that improperly handle the deployment graph in case of affected nodes.