do-release-upgrade races puppet for file contents

[Nick Moffitt]

This is something of a larger problem with configuration management, but if puppet is running while do-release-upgrade runs, puppet may re-assert contents of files that the package manager has just changed.

Puppet allows you to key off of the currently running distro release, so one could imagine boot-critical configuration (filesystems, etc) being set to e.g. Lucid versions just before a reboot into Precise. If the reboot succeeds, puppet would subsequently deliver the Precise versions and things could sort themselves out, but the reboot could well be stymied by older Lucid versions.

The canonical-memento puppet manifests include a lot of safety features to prevent precisely this scenario: we insist that /etc be maintained in bzr, and we halt all configuration management if there are uncommitted changes there. This means that once the package manager changes any file in /etc, puppet refuses to run. Not everyone will be as careful as we are, and even our system is not atomic.

This is of course not specific to do-release-upgrade, and it's a problem any time you upgrade a package. It's just most dangerous during d-r-u, because of the breadth and depth of changes.

The upgrade could shut down the puppet agent before upgrading, but once puppet itself is upgraded that will trigger a restart. It's probably best to do `puppet agent --disable` before shutting down puppet. Once this is done, it may be worth warning the sysadmin at the end (perhaps while advising a reboot) that puppet was running before the upgrade, and that something like `puppet agent -t --noop` is useful for seeing what changes puppet would make.

Of course, it's an architectural flaw in puppet that --noop runs still upload exported resources to the database, and that's just a hazard of puppet administration these days.

[Adam Conrad (adconrad)]

It seems that this isn't really an update-manager (or package management in general) issue at all, but rather puppet should be shipping an apt.conf.d snippet with DPkg::Pre-Invoke and DPkg::Post-Invoke hooks to stop and start the puppet daemon before and after any operations are done.

[Adam Conrad (adconrad)]

Note that the above fix should solve any package management versus puppet concurrency issues (from apt all the way up to update-manager) except for the case of admins running dpkg by hand on individual debs, but one would hope the admin has a fairly good idea of their expected outcome in that case.

[Nick Moffitt (nick-moffitt)]

That could work, provided that these hooks allow for the situation where we don't want puppet to start up immediately after the upgrade has completed.

[Serge Hallyn (serge-hallyn)]

Marking high priority because while it may be uncommon and unlikely, it could definately impact core applications.

[Robie Basak (racb)]

What will happen when puppet runs "apt-get install" because a manifest requires certain packages to be present?

[Clint Byrum (clint-fewbar)]

It seems to me that this is a locking issue, and Adam's solution should be tweaked a bit to have apt register some kind of lock system whereby only apt, or puppet, can be running at the same time.

Anyway, I'm moving this to Confirmed. It should probably be forwarded to puppet upstream for their consideration as well.