[astute] If puppet fails with dependency graph, fuel can't determine it

Bug #1326583 reported by Andrey Grebennikov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Vladimir Sharshov

Bug Description

If I change manifests and make a mistake, or in any other case if the deployment failed on the node with multiple roles assigned, puppet returns 0 as an exit code. Consequently Fuel continue deployment specifying the next assigned role to the node.
Seems something has been fixed in puppet.

Tags: astute
Revision history for this message
Andrew Woodward (xarses) wrote :

This usually happens when puppet is evaluating the dependency graph and dies due for some reason, In this case puppet will return 0 and astute thinks it done. This is confirmed on puppet 2.7. We need to verify if this applies to puppet 3.4, If so it needs to be accommodated for.

Changed in fuel:
status: New → Confirmed
Revision history for this message
Andrey Grebennikov (agrebennikov) wrote :
Mike Scherbakov (mihgen)
no longer affects: fuel/4.1.x
no longer affects: fuel
Mike Scherbakov (mihgen)
summary: - Fuel continues deploy next role if previous failed
+ If puppet fails with dependency graph, fuel can't determine it
Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/5.0.x
Changed in fuel:
status: New → Confirmed
importance: Undecided → Medium
assignee: nobody → Fuel Astute Team (fuel-astute)
milestone: none → 5.1
no longer affects: mos
Revision history for this message
Vladimir Sharshov (vsharshov) wrote : Re: If puppet fails with dependency graph, fuel can't determine it

We run puppet using daemon and we could not get error code about catalog. We can create special check which would run before real puppet run and prevent running if we got error.

Command for check: puppet apply --noop /etc/puppet/manifests/site.pp

Example:
[root@node-5 ~]# puppet apply --noop /etc/puppet/manifests/site.pp
Error: Duplicate declaration: Stage[zero] is already declared in file /etc/puppet/manifests/site.pp:15; cannot redeclare at /etc/puppet/manifests/site.pp:20 on node node-5.domain.tld
Error: Duplicate declaration: Stage[zero] is already declared in file /etc/puppet/manifests/site.pp:15; cannot redeclare at /etc/puppet/manifests/site.pp:20 on node node-5.domain.tld
[root@node-5 ~]# echo $?
1

It can take perceptible time (3-5 minutes for controller node).

Revision history for this message
Vladimir Sharshov (vsharshov) wrote :

Problem more complicated. --noop return bad code if catalog compiled succeed, but process emulation failed because required bin files missed.

Revision history for this message
Dmitry Ilyin (idv1985) wrote :

We need to add --only-catalog option for puppet apply

summary: - If puppet fails with dependency graph, fuel can't determine it
+ [astute] If puppet fails with dependency graph, fuel can't determine it
Changed in fuel:
milestone: 5.1 → 6.0
Changed in fuel:
assignee: Fuel Astute Team (fuel-astute) → Vladimir Sharshov (vsharshov)
tags: added: astute
Revision history for this message
Vladimir Sharshov (vsharshov) wrote :
Changed in fuel:
assignee: Vladimir Sharshov (vsharshov) → Fuel OSCI Team (fuel-osci)
Revision history for this message
Vladimir Sharshov (vsharshov) wrote :

Osci guys, please create separate repo and include this patch for puppet 3.4.2 for Ubuntu and Centos client nodes

Projects:
- openstack-ci/precise/puppet
- openstack-ci/centos64/puppet

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package puppet has been built from changeset: http://gerrit.mirantis.com/21248
RPM Repository URL: http:///osci-obs.vm.mirantis.net:82/centos-fuel-master-21248/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package puppet has been built from changeset: http://gerrit.mirantis.com/21250
DEB Repository URL: http:///osci-obs.vm.mirantis.net:82/ubuntu-fuel-master-21250/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package puppet has been built from changeset: http://gerrit.mirantis.com/21248
RPM Repository URL: http:///osci-obs.vm.mirantis.net:82/centos-fuel-master-21248/centos

Roman Vyalov (r0mikiam)
Changed in fuel:
assignee: Fuel OSCI Team (fuel-osci) → Vladimir Sharshov (vsharshov)
Changed in fuel:
milestone: 6.0 → 6.1
Revision history for this message
Dmitry Ilyin (idv1985) wrote :

POrting this algorith to puppetd mco agent will help too https://review.openstack.org/#/c/130286/5/tasklib/tasklib/actions/puppet.py,cm

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package puppet has been built for project packages/precise/puppet
Package version == 3.4.2, package release == ubuntu1

Changeset: https://review.fuel-infra.org/1300
project: packages/precise/puppet
branch: 6.0
author: Vladimir Kuklin
committer: Vladimir Kuklin
subject: Apply patch catalog-compilation-exit-code.patch to v.3.4.2
status: patchset-created

Files placed on repository:
puppet-common_3.4.2-ubuntu1_all.deb
puppet-el_3.4.2-ubuntu1_all.deb
puppet-testsuite_3.4.2-ubuntu1_all.deb
puppet_3.4.2-ubuntu1_all.deb
puppetmaster-common_3.4.2-ubuntu1_all.deb
puppetmaster-passenger_3.4.2-ubuntu1_all.deb
puppetmaster_3.4.2-ubuntu1_all.deb
vim-puppet_3.4.2-ubuntu1_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable-1300/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package puppet has been built for project packages/centos6/puppet
Package version == 3.4.2, package release == 2

Changeset: https://review.fuel-infra.org/1298
project: packages/centos6/puppet
branch: 6.0
author: Vladimir Kuklin
committer: Vladimir Kuklin
subject: Apply patch catalog-compilation-exit-code.patch to v.3.4.2
status: patchset-created

Files placed on repository:
puppet-3.4.2-2.mira3.noarch.rpm
puppet-server-3.4.2-2.mira3.noarch.rpm

NOTE: Changeset is not merged, created temporary package repository.
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable-1298/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

RPM package puppet has been built for project packages/centos6/puppet
Package version == 3.4.2, package release == 2

Changeset: https://review.fuel-infra.org/1298
project: packages/centos6/puppet
branch: 6.0
author: Vladimir Kuklin
committer: Vladimir Kuklin
subject: Apply patch catalog-compilation-exit-code.patch to v.3.4.2
status: change-merged

Files placed on repository:
puppet-3.4.2-2.mira3.noarch.rpm
puppet-server-3.4.2-2.mira3.noarch.rpm

Changeset merged. Package placed on primary repository
RPM repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-6.0-stable/centos

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package puppet has been built for project packages/precise/puppet
Package version == 3.4.2, package release == ubuntu1

Changeset: https://review.fuel-infra.org/1300
project: packages/precise/puppet
branch: 6.0
author: Vladimir Kuklin
committer: Vladimir Kuklin
subject: Apply patch catalog-compilation-exit-code.patch to v.3.4.2
status: change-merged

Files placed on repository:
puppet-common_3.4.2-ubuntu1_all.deb
puppet-el_3.4.2-ubuntu1_all.deb
puppet-testsuite_3.4.2-ubuntu1_all.deb
puppet_3.4.2-ubuntu1_all.deb
puppetmaster-common_3.4.2-ubuntu1_all.deb
puppetmaster-passenger_3.4.2-ubuntu1_all.deb
puppetmaster_3.4.2-ubuntu1_all.deb
vim-puppet_3.4.2-ubuntu1_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-6.0-stable/ubuntu

Changed in fuel:
milestone: 6.1 → 6.0
importance: Medium → Critical
status: Confirmed → Triaged
Changed in fuel:
status: Triaged → Fix Committed
tags: added: on-verification
Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: on-verification
Revision history for this message
Anastasia Palkina (apalkina) wrote :

There is old package on ISO #49

-rwxr-xr-x. 1 root root 9610 Дек 16 10:17 puppet_3.4.2-1puppetlabs1_all.deb

Need to verify on new RC

Changed in fuel:
status: Fix Released → Fix Committed
tags: added: on-verification
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified on ISO #56

"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"

Changed in fuel:
status: Fix Committed → Fix Released
tags: removed: on-verification
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.