| 2016-07-27 09:01:55 |
Sergey Novikov |
bug |
|
|
added bug |
| 2016-07-27 09:02:34 |
Sergey Novikov |
fuel: importance |
Undecided |
High |
|
| 2016-07-27 09:21:07 |
Vladimir Kuklin |
description |
Detailed bug description:
After the upgrade of Fuel master node from 8 to 9.1 with the existing 8.0 cluster and the further scaling this cluster fails with error in the puppet's log:
Could not run: Could not find file /etc/puppet/modules/osnailyfacter/modular/hiera/hiera.pp
The root cause is task "rsync_core_puppet" doesn't execute and manifests are not delivered to the deploying node
Steps to reproduce:
1. Deploy 8.0 Fuel node
2. Create cluster with 3 cntrl + 3 compute+ceph-osd
3. Upgrade Fuel master node from 8.0 to 9.1 (http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-install-guide/upgrade/upgrade-fuel.html)
4. Add node with role "compute" to created cluster
5. Deploy changes
Octane's versions:
8.0.0-1.mos1192
9.0.0-1.mos1208
ISO's versions:
8.0 - http://paste.openstack.org/show/538992/
9.1 - http://paste.openstack.org/show/542518/ |
Detailed bug description:
After the upgrade of Fuel master node from 8 to 9.1 with the existing 8.0 cluster and the further scaling this cluster fails with error in the puppet's log:
Could not run: Could not find file /etc/puppet/modules/osnailyfacter/modular/hiera/hiera.pp
The root cause is task "rsync_core_puppet" doesn't execute and manifests are not delivered to the deploying node
Steps to reproduce:
1. Deploy 8.0 Fuel node
2. Create cluster with 3 cntrl + 3 compute+ceph-osd
3. Upgrade Fuel master node from 8.0 to 9.1 (http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-install-guide/upgrade/upgrade-fuel.html)
4. Add node with role "compute" to created cluster
5. Deploy changes
Expected result:
Deployment succeeds
Actual result:
Deployment fails as rsync_core_puppet task does not run on the newly added node.
This happens due to the fact that pre_|post_deployment stages do not contain any tasks. Here is a excerpt of debugged print from task.py for granular_deploy:
'post_deployment': [],
'pre_deployment': []
What's also important that the deployment succeeds if we provision the node first and deploy it then.
Octane's versions:
8.0.0-1.mos1192
9.0.0-1.mos1208
ISO's versions:
8.0 - http://paste.openstack.org/show/538992/
9.1 - http://paste.openstack.org/show/542518/ |
|
| 2016-07-27 09:50:24 |
Ilya Kharin |
fuel: status |
New |
Confirmed |
|
| 2016-07-27 13:08:30 |
Alexander Kislitsky |
fuel: assignee |
|
Fuel Octane (fuel-octane-team) |
|
| 2016-08-10 10:55:41 |
Sergey Shevorakov |
tags |
|
blocker-for-qa |
|
| 2016-08-12 21:09:37 |
Ilya Kharin |
fuel: assignee |
Fuel Octane (fuel-octane-team) |
Ilya Kharin (akscram) |
|
| 2016-08-19 06:53:21 |
Ilya Kharin |
description |
Detailed bug description:
After the upgrade of Fuel master node from 8 to 9.1 with the existing 8.0 cluster and the further scaling this cluster fails with error in the puppet's log:
Could not run: Could not find file /etc/puppet/modules/osnailyfacter/modular/hiera/hiera.pp
The root cause is task "rsync_core_puppet" doesn't execute and manifests are not delivered to the deploying node
Steps to reproduce:
1. Deploy 8.0 Fuel node
2. Create cluster with 3 cntrl + 3 compute+ceph-osd
3. Upgrade Fuel master node from 8.0 to 9.1 (http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-install-guide/upgrade/upgrade-fuel.html)
4. Add node with role "compute" to created cluster
5. Deploy changes
Expected result:
Deployment succeeds
Actual result:
Deployment fails as rsync_core_puppet task does not run on the newly added node.
This happens due to the fact that pre_|post_deployment stages do not contain any tasks. Here is a excerpt of debugged print from task.py for granular_deploy:
'post_deployment': [],
'pre_deployment': []
What's also important that the deployment succeeds if we provision the node first and deploy it then.
Octane's versions:
8.0.0-1.mos1192
9.0.0-1.mos1208
ISO's versions:
8.0 - http://paste.openstack.org/show/538992/
9.1 - http://paste.openstack.org/show/542518/ |
Detailed bug description:
After the upgrade of Fuel master node from 8 to 9.1 with the existing 8.0 cluster and the further scaling this cluster fails with error in the puppet's log:
Could not run: Could not find file /etc/puppet/modules/osnailyfacter/modular/hiera/hiera.pp
This happens because of
The root cause is task "rsync_core_puppet" doesn't execute and manifests are not delivered to the deploying node
Was found that granular_deploy modifies graph in case of some nodes are affected by deployment (e.g. in case of scaling a cluster):
2016-08-16 09:00:31.242 DEBUG [7f0057829880] (manager) There are nodes to deploy: node-9.test.domain.local
2016-08-16 09:00:31.285 DEBUG [7f0057829880] (manager) There are nodes affected by deployment: node-7.test.domain.local node-4.test.domain.local node-5.test.domain.local node-8.test.domain.local node-
2.test.domain.local node-6.test.domain.local node-3.test.domain.local node-1.test.domain.local
and this modification of the graph makes all non-reexecutable tasks as `skipped`:
...
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task rsync_core_puppet will be skipped.
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task clear_nodes_info will be skipped.
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task generate_keys will be skipped.
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task copy_keys will be skipped.
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task generate_haproxy_keys will be skipped.
2016-08-16 14:59:03.297 DEBUG [7f0057829880] (orchestrator_graph) Task copy_haproxy_keys will be skipped.
2016-08-16 14:59:03.298 DEBUG [7f0057829880] (orchestrator_graph) Task sync_time will be skipped.
...
after that the graph is modified to handle only affected nodes and is incorrect for nodes that have to be deployed.
Steps to reproduce:
1. Deploy 8.0 Fuel node
2. Create cluster with 3 cntrl + 3 compute+ceph-osd
3. Upgrade Fuel master node from 8.0 to 9.1 (http://docs.openstack.org/developer/fuel-docs/userdocs/fuel-install-guide/upgrade/upgrade-fuel.html)
4. Add node with role "compute" to created cluster
5. Deploy changes
Expected result:
Deployment succeeds
Actual result:
Deployment fails as rsync_core_puppet task does not run on the newly added node.
This happens due to the fact that pre_|post_deployment stages do not contain any tasks. Here is a excerpt of debugged print from task.py for granular_deploy:
'post_deployment': [],
'pre_deployment': []
What's also important that the deployment succeeds if we provision the node first and deploy it then.
Workaround:
Some modifications can be done to have two graphs but it leads to run whole deployment graph on affected nodes too:
diff --git a/nailgun/nailgun/task/task.py b/nailgun/nailgun/task/task.py
index b5f3b6a..e985652 100644
--- a/nailgun/nailgun/task/task.py
+++ b/nailgun/nailgun/task/task.py
@@ -330,9 +330,10 @@ class DeploymentTask(BaseDeploymentTask):
cls._save_deployment_info(transaction, serialized_cluster)
if affected_nodes:
- graph.reexecutable_tasks(events)
+ reexec_graph = graph.copy()
+ reexec_graph.reexecutable_tasks(events)
serialized_cluster.extend(deployment_serializers.serialize(
- graph, transaction.cluster, affected_nodes
+ reexec_graph, transaction.cluster, affected_nodes
))
nodes = nodes + affected_nodes
pre_deployment = stages.pre_deployment_serialize(
Octane's versions:
8.0.0-1.mos1192
9.0.0-1.mos1208
ISO's versions:
8.0 - http://paste.openstack.org/show/538992/
9.1 - http://paste.openstack.org/show/542518/ |
|
| 2016-08-19 06:53:34 |
Ilya Kharin |
fuel: assignee |
Ilya Kharin (akscram) |
Fuel Sustaining (fuel-sustaining-team) |
|
| 2016-08-19 07:18:32 |
Maksim Malchuk |
tags |
blocker-for-qa |
area-python blocker-for-qa |
|
| 2016-08-19 10:24:16 |
Georgy Kibardin |
fuel: assignee |
Fuel Sustaining (fuel-sustaining-team) |
Dmitry Guryanov (dguryanov) |
|
| 2016-08-19 11:00:38 |
Dmitry Pyzhov |
nominated for series |
|
fuel/mitaka |
|
| 2016-08-19 11:00:38 |
Dmitry Pyzhov |
bug task added |
|
fuel/mitaka |
|
| 2016-08-19 11:00:44 |
Dmitry Pyzhov |
fuel: milestone |
9.1 |
10.0 |
|
| 2016-08-19 11:00:46 |
Dmitry Pyzhov |
fuel/mitaka: milestone |
|
9.1 |
|
| 2016-08-19 11:00:53 |
Dmitry Pyzhov |
fuel/mitaka: assignee |
|
Dmitry Guryanov (dguryanov) |
|
| 2016-08-19 11:00:55 |
Dmitry Pyzhov |
fuel/mitaka: status |
New |
Confirmed |
|
| 2016-08-19 11:00:58 |
Dmitry Pyzhov |
fuel/mitaka: importance |
Undecided |
High |
|
| 2016-08-30 07:44:23 |
OpenStack Infra |
fuel: status |
Confirmed |
In Progress |
|
| 2016-09-07 12:57:40 |
OpenStack Infra |
fuel: status |
In Progress |
Fix Committed |
|
| 2016-09-07 14:53:01 |
OpenStack Infra |
fuel/mitaka: status |
Confirmed |
In Progress |
|
| 2016-09-08 10:13:03 |
OpenStack Infra |
fuel/mitaka: status |
In Progress |
Fix Committed |
|
| 2016-09-09 12:24:10 |
Vladimir Khlyunev |
fuel/mitaka: status |
Fix Committed |
Fix Released |
|
| 2016-09-13 15:04:22 |
Vladimir Khlyunev |
fuel/mitaka: status |
Fix Released |
In Progress |
|
| 2016-09-14 10:20:02 |
OpenStack Infra |
fuel/mitaka: status |
In Progress |
Fix Committed |
|
| 2016-09-15 10:04:49 |
Vladimir Khlyunev |
fuel/mitaka: status |
Fix Committed |
Fix Released |
|
| 2017-02-09 07:50:43 |
OpenStack Infra |
tags |
area-python blocker-for-qa |
area-python blocker-for-qa in-stable-newton |
|
