puppet uses upstart for service status checks in vivid

[Tom Hendrikx]

After upgrading one of my servers to 15.04/vivid (from original 14.10/utopic install), I had a failing puppet job. The error message is:

Error: /Stage[main]/Main/Service[ssh]: Could not evaluate: Execution of '/sbin/status ssh' returned 1: status: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

After some research it seems that this happens with all daemons that have an upstart config file hanging around in '/etc/init/'. Puppet checks for existence of that file, and then assumes that it should issue the '/sbin/status <service>' command in order to check the service status. For all services that don't have an upstart conf installed, puppet uses '/etc/init.d/<service> status', which works perfectly (for ssh and munin-node, the two services I used while verifying the behavior).

As I assume that we cannot expect to have all services drop the upstart config in their vivid packages (I'm not sure about the status of upstart support in vivid), the best fix should be to have puppet check for '/etc/init.d/<service>' before trying '/etc/init/<service>/.conf', at least on vivid.

puppet-common version: 3.7.2-1ubuntu2

Note: even setting service 'hasstatus => no' and 'status => /etc/init.d/ssh status' in the puppet Service definition does not force puppet to not issue the command (which probably should be a separate bug).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robie Basak (racb)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

This sounds like an upstream puppet "bug" because Ubuntu follows the Debian design in the new world order. In Debian one might switch init systems at will, so the existence of service files in /etc/init/ or /etc/init.d/ or /{etc,lib}/systemd/system/ cannot really determine anything. Instead, one has to pay attention to which init system is active (I'm not sure what the "official" way is to do that, but we can find that out).

Have upstream tackled this yet at all? If not, then I think it would be appropriate to file a bug there. Once upstream have settled on a fix, we can cherry-pick it to Vivid.

[Robie Basak (racb)]

Or perhaps puppet should exclusively be using the "service" wrapper now (/usr/sbin/service), certainly on Ubuntu and presumably on Debian?

[Martin Pitt (pitti)]

> Instead, one has to pay attention to which init system is active (I'm not sure what the "official" way is to do that, but we can find that out).

Right. Checking files is *never* correct, as both in upstart and in systemd you can always override them (e. g. /etc/init/*.override or a file in /etc/systemd/system/ or /run/systemd/system/ overrides a file in /lib/systemd/system).

> Or perhaps puppet should exclusively be using the "service" wrapper now

That is the correct thing to do. service works under SysV, upstart, and systemd. The only thing to keep in mind is that "service status foo" always exits with 0 under upstart even if a service isn't running. Arguably this really ought to be fixed, so if you need that functionality let's fix that properly instead of working around it.

[Ben Shephard (ben-shephard)]

I've just tested this on a fresh build of 15.04 and it seems to work just fine so I think the issue stems from the upgrade not changing the provider for service management from upstart to systemd

[Robie Basak (racb)]

On Tue, May 26, 2015 at 03:16:04PM -0000, Ben Shephard wrote:
> ...I think the issue stems from the upgrade not changing the
> provider for service management from upstart to systemd

I thought the providers are dynamically determined by puppet at runtime
and not stored persistently anywhere?

[Stig Sandbeck Mathisen (ssm)]

Correct behaviour (default to systemd, determine if it is running, fallback to upstart / sysvinit) seems to have been added to puppet upstream at 4.1.0-87-g7fe6164, just a few days ago. This looks like it will be part of the puppet 4.1.1 release.

The packages currently in debian stable was changed to use the "service" command, which should work with all init systems in Debian/Ubuntu at this time.

The upstream changes look possible to backport to puppet 3.7.

[Robie Basak (racb)]

Thanks Stig. Looks like we want to cherry-pick relevant changes between 3.7.2-1 and 3.7.2-4.

[Robie Basak (racb)]

Is this related to bug 1495853?

[Michael Wodniok (damihe)]

This still exists in current dev-version of Ubuntu 15.10. Is this a bug caused by the (quite old, current Puppet-version is 4.2.2) Ubuntu package or by puppet itself?

[Hadmut Danisch (hadmut)]

I also ran into that problem on 15.10 beta.

[Oliver Chick (oc243)]

Experiencing this on 15.10

[AmirAli Akbari (amiraliakbari)]

Experiencing after update to 15.10

As upstart is replaced by systemd, I changed the /sbin/status to use the service command:

    #!/bin/bash
    /usr/sbin/service $1 status
    echo $?

which solves the problem, just mentioned it if anyone wants to get puppet running anyway.

[Christian Ehrhardt  (paelzer)]

Recent merges have "somewhat" fixed this issue, but only if upstart is not installed.
And @Robie - yes it was related to bug 1495853

Tested in Wily&Xenial containers with:
apt-get install openssh-server
apt-get install puppet
puppet resource --debug --param provider --param enable service ssh
puppet resource --debug --param provider service ssh ensure=stopped
puppet resource --debug --param provider service ssh enable=false
puppet resource --debug --param provider service ssh enable=true

Just one snippet of these how things now work:
puppet resource --debug --param provider service ssh enable=true
Debug: Runtime environment: puppet_version=3.7.2, ruby_version=2.1.5, run_mode=user, default_encoding=US-ASCII
Debug: Puppet::Type::Service::ProviderRedhat: file /sbin/chkconfig does not exist
Debug: Puppet::Type::Service::ProviderDaemontools: file /usr/bin/svc does not exist
Debug: Puppet::Type::Service::ProviderRunit: file /usr/bin/sv does not exist
Debug: Puppet::Type::Service::ProviderGentoo: file /sbin/rc-update does not exist
Debug: Puppet::Type::Service::ProviderLaunchd: file /bin/launchctl does not exist
Debug: Puppet::Type::Service::ProviderOpenrc: file /bin/rc-status does not exist
Debug: Puppet::Type::Service::ProviderUpstart: file /sbin/start does not exist
Debug: Loaded state in 0.00 seconds
Debug: Executing '/usr/sbin/service ssh status'
Debug: Executing '/bin/systemctl show -pSourcePath ssh'
Debug: Executing '/bin/systemctl is-enabled ssh'
Debug: Finishing transaction 27170800
Debug: Storing state
Debug: Stored state in 0.01 seconds
Debug: Executing '/usr/sbin/service ssh status'
Debug: Executing '/bin/systemctl show -pSourcePath ssh'
Debug: Executing '/bin/systemctl is-enabled ssh'
service { 'ssh':
  ensure => 'running',
  enable => 'true',
  provider => 'debian',
}

As pitti suggested it uses the service wrapper now.

To be 100% sure that we have no issue with an 14.04->16.04 upgrade path leaving stale files that might affect this I tested that as well.
One could nicely see that before the upgrade it was upstart
[...]
Debug: Executing '/sbin/status ssh'
[...]
  provider => 'upstart',

$ do-release-upgrade -d
+ Same puppet commands again

There things got interesting.
While still having the new puppet version it is now running into mentioned issues:
Debug: Executing '/sbin/status ssh'
Error: /Service[ssh]: Could not evaluate: Execution of '/sbin/status ssh' returned 1: status: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused

The difference seems to be the existance of /sbin/start in the system that got upgraded 14.04 -> 16.04.
The clean 16.04 has:
Debug: Puppet::Type::Service::ProviderUpstart: file /sbin/start does not exist
But the upgraded system has upstart installed, and due to that runs into the issues.
Just having it installed does not mean it IS the current init system.

Removing upstart on the 14.04->16.04 upgraded system resolves the situation.
It detects it as provider => debian then and works.
Vice versa if you install the upstart package on the clean 16.04 it runs into the same issues.

So TL;DR
- puppets detection which provider to use seems to base on file checking...

[david.barbion (david-barbion)]

Christian, how did you manage to uninstall upstart package without breaking the whole system?
On my updated systems, when I try to uninstall upstart, apt-get says it have to remove ubuntu-desktop, unity and unity-greeter:

root@xxxx:~# LANG=C apt-get purge upstart
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  ubuntu-desktop* unity* unity-greeter* upstart*
0 upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
6 not fully installed or removed.
After this operation, 9 203 kB disk space will be freed.
Do you want to continue? [Y/n]

David.

[Sledge Sulaweyo (sledge-sulaweyo)]

Xenial is affected by that as well -> standard setup and puppet service calls fail

[ghomem (gustavo)]

This affects 16.04 desktop.

I'm using a KDE based version, without unity, etc and I managed to remove upstart without dependency problems - simply ran "apt-get remove upstart".

[Mathew Hodson (mhodson)]

Is this the same as Bug #1570472 ?

That bug has a patch that will make systemd the default if it is running.

[ghomem (gustavo)]

@mathey hodson

yes ,this bug is the same as Bug #1570472.

[ghomem (gustavo)]

I confirm that

puppet-3.8.5-2

from this PPA

https://launchpad.net/~nacc/+archive/ubuntu/lp1570472

resolves the problem, at least on Ubuntu 16.04 x64.

This was suggested on Bug #1570472. Thanks Mathew Hodson for pointing out.

[llbbl (llbbl)]

I can also confirm that PPA @gustavo linked to works for that puppet + ubuntu version.

Thanks for the fix. Ok now back to puppet'ing.

L