Possible puppet performance regression with 2.7.11-1ubuntu2.4

[Brad Marshall]

We appear to have a performance regression with puppet 2.7.11-1ubuntu2.4 that we recently upgraded to, particularly on our more heavily loaded puppet master. When we're running 2.4, many of our puppet clients get the following:

err: Could not retrieve catalog from remote server: execution expired

and the load on the puppet master is higher than with 2.3. When we revert back to 2.3, the load is much lower (around 20 rather than around 40), and most of the puppet clients can retrieve the catalog without a problem.

Is there any further information you need, or any debugging we can assist with finding out what the issue is here?

OS Version: Ubuntu 12.04.3 LTS

$ dpkg-query -W puppet
puppet 2.7.11-1ubuntu2.4

[Matthaus Owens (matthaus-m)]

This is probably related to upstream bug https://projects.puppetlabs.com/issues/21337

[Robie Basak (racb)]

Thanks for finding the upstream bug. But it looks like upstream is talking about a regression introduced by switching to safe_yaml. In Ubuntu, the safe_yaml module change happened in 2.7.11-1ubuntu2.3, not 2.7.11-1ubuntu2.4.

It looks like 2.7.11-1ubuntu2.4 included a roll-up of various different patches. To debug, perhaps we can bisect this? https://github.com/puppetlabs/puppet/commits/2.7.x might be a source of the individual patches - but they don't seem to line up exactly.