Supervisor not enabled or started in Ubuntu 16.04 after installation

Bug #1594740 reported by Peter Odding on 2016-06-21
162
This bug affects 33 people
Affects Status Importance Assigned to Milestone
supervisor (Ubuntu)
Medium
Unassigned
Xenial
Undecided
Nish Aravamudan
Yakkety
Undecided
Unassigned

Bug Description

[Impact]

 * In all prior releases to 16.04, `apt-get install supervisor` would result in supervisor being started after package installation (and configured to start by default at boot). This is the principle of least surprise in action and reflects Ubuntu's ease of use.

 * Due to the timing of 16.04, the version from Debian was broken to not support systemd properly.

 * Both Debian and Ubuntu have been fixed since the version in 16.04, in the same way as the proposed fix.

[Test Case]

 * Install supervisor. After installation, the service should be running and should be enabled.

[Regression Potential]

 * As the behavior is currently regressed relative to 14.04, I believe the primary potential source of regressions will be users who have manually fixed-up systemd to do what was expected on their systems. I believe dpkg will notice the differences, if any, between the system-installed supervisor systemd files and those in this package.

---

Expected behavior
=================

In Ubuntu 10.04, 12.04 and 14.04 after running "apt-get install supervisor" the Supervisor daemon is automatically enabled (to start on boot) and started (so that Supervisor is running by the time apt-get returns).

What actually happens
=====================

In Ubuntu 16.04 the Supervisor daemon is not automatically enabled nor is it started during installation. This breaks compatibility with previous and expected behavior.

Why this is a problem
=====================

I've built dozens of tools that use Supervisor for process supervision and these tools are deployed using custom Debian packages. Each of these packages has a dependency on the supervisor package with the expectation that Supervisor will be installed, enabled and started so that my post-installation scripts can call "supervisorctl" and expect it to work (instead of complaining about a missing UNIX socket file and exiting with a nonzero status code, thereby breaking my automated server provisioning).

Known workaround
================

Create a shim package with a dependency on supervisor and a post-installation script that runs the following commands:

    # On Ubuntu 16.04 the installation of Supervisor does not
    # enable and start the Supervisor daemon which breaks
    # compatibility with previous Ubuntu releases.
    if [ $(lsb_release --short --codename) = xenial ]; then
      # Make sure the daemon is enabled.
      if ! systemctl --quiet is-enabled supervisor; then
        systemctl enable supervisor
      fi
      # Make sure the daemon is started.
      if ! systemctl --quiet is-active supervisor; then
        systemctl start supervisor
      fi
    fi

Alternatively one can obviously just run these commands by hand to rectify the situation.

It's kind of nasty that I have to create a shim package like this to compensate for a break in backwards compatibility that is -as far as I know- undocumented and most likely unintentional. What's more is that a lot of people will lack the means to create shim packages like this, so thousands of Ubuntu users / integrators / system administrators worldwide will need to repeat these shenanigans manually.

Affected versions
=================

    peter@template-xenial:~$ lsb_release --short --description
    Ubuntu 16.04 LTS

    peter@template-xenial:~$ apt-cache policy supervisor
    supervisor:
      Installed: 3.2.0-2
      Candidate: 3.2.0-2
      Version table:
     *** 3.2.0-2 500
            500 http://mirror.nl.leaseweb.net/ubuntu xenial/universe amd64 Packages
            500 http://mirror.nl.leaseweb.net/ubuntu xenial/universe i386 Packages
            100 /var/lib/dpkg/status

Root cause analysis
===================

In Ubuntu 16.04 Supervisor is managed by systemd however the post-installation script is using update-rc.d and invoke-rc.d to enable and start Supervisor. As far as I know these commands are remnants of the old daemon management infrastructure and they don't integrate with systemd, hence the breakage:

    peter@template-xenial:~$ cat /var/lib/dpkg/info/supervisor.postinst
    #!/bin/sh
    set -e

    # Automatically added by dhpython:
    if which pycompile >/dev/null 2>&1; then
      pycompile -p supervisor
    fi

    # End automatically added section
    # Automatically added by dh_installinit
    if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ]; then
      if [ -x "/etc/init.d/supervisor" ]; then
        update-rc.d supervisor defaults >/dev/null
      fi
      if [ -x "/etc/init.d/supervisor" ] || [ -e "/etc/init/supervisor.conf" ]; then
        invoke-rc.d supervisor start || exit $?
      fi
    fi
    # End automatically added section

Conclusion
==========

Is there a remote chance of getting this fixed in Ubuntu 16.04, maybe in a later point release? On the one hand I get that fixing this now is a big change compared to the original release of Ubuntu 16.04, on the other hand I have found dozens of unsuspecting users (see below) being bitten by this change in behavior and I haven't found a single user who appreciated it :-).

If there is no chance of fixing this it should at least be documented in the release notes as a known regression, because I haven't been able to find any proper documentation about this change and I have found dozens of people being bitten by the break in backwards compatibility.

External references
===================

Some random reports of people running into (what I believe is) this exact issue:

Here's an upstream bug report, where nothing can be fixed:
https://github.com/Supervisor/supervisor/issues/735

Here's an unhappy user wondering how to restore expected behavior:
http://unix.stackexchange.com/questions/281774/ubuntu-server-16-04-cannot-get-supervisor-to-start-automatically

information type: Public → Public Security
information type: Public Security → Public
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in supervisor (Ubuntu):
status: New → Confirmed
Changed in supervisor (Ubuntu):
importance: Undecided → Medium
Aditya (code-aditya) wrote :

Please fix this regression. I just used Ubuntu 16.04, now that the first point release is out, for one of my new projects and was immediately impacted by this bug. Took almost an hour for me to figure out the problem since I didn't have to configure this on earlier releases. More people would become aware of this issue now since 16.04.1 is out.

Alexis de Lattre (alexis-via) wrote :

Thanks Peter for this very detailed bug report ! I am also impacted by this annoying bug :-( Hope it gets fixed soon.

Ken S (kas130771) wrote :

Thank you for putting together this report. Also impacted by this bug..

Miha Vrhovnik (mvrhov) wrote :

I've also upgraded 14.04 to 16.04 yesterday and have been bitten by this bug.

Craig Kelly (craig-n-kelly) wrote :

I've also been impacted by this bug in 16.04

Cesc Fran (fran325) wrote :

Also affected by this behaviour of supervisor, as installing erpnext (discuss.erpnext.com) version 7, on ubuntu 16.04, and it took me quite some time to find out that supervisor was not started at boot time ... Hope that will be taken into account ...

thanks Peter for the explanation !

FR

Chester (cd-pobox) wrote :

Also have this problem with a fresh install of 16.04. Thank you for the bug report, and hope it gets fixed soon.

Orestis Ioannou (orestis-8) wrote :

Hello,

I maintain supervisor in Debian and I am really sorry for the trouble i caused here. I didn't know that i had to build-depend on dh-systemd and change d/rules for the package to be enabled since for sys-v there is no such thing needed.

I received a bug report in Debian but didn't think it was urgent because i hadn't looked here as well :/ Again really sorry for the trouble i caused. I uploaded a new version of supervisor and fixed this issue. I am not sure what i can do more though for ubuntu 16.04. I am open to suggestions for anyone familiar with ubuntu proceedings.

Cheers,

Orestis

Peter Odding (peterodding) wrote :

Hi Orestis and thanks for following up here even though you're a _Debian_ package maintainer and not an _Ubuntu_ package maintainer :-).

Am I assuming correctly that the Debian bug report you are referring to is the following? https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=827729

I will investigate tomorrow morning whether there is any chance of getting your fix into Ubuntu 16.04. Like you I am not intimately familiar with the Ubuntu release management process, it's not even clear to me whether non-security bug fixes can be backported to LTS releases. I suppose they _might_ if 1) there are no backwards incompatible changes, 2) the bug is considered critical enough and 3) enough people make their voice heard that they consider it a critical bug :-).

Will follow up here soon!

Peter Odding (peterodding) wrote :

I contacted <email address hidden> to inquire whether it's possible to get the bug fix in the Debian package backported to Ubuntu 16.04, here's my message:

https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2016-September/016866.html

Nish Aravamudan (nacc) wrote :

Thanks for filing this bug report!

A backport seems reasonable. Please follow https://wiki.ubuntu.com/StableReleaseUpdates, in particular the procedure section.

Nish Aravamudan (nacc) wrote :

Note that probably this will need fixing first in 16.10 as the same version is in both releases.

Orestis Ioannou (orestis-8) wrote :

Yes that is the correct one.
I am short on time these days and can't help with backporting the package for ubuntu. Would you be able to take care of this ?
If you only want to backport the systemd change here's what you need https://anonscm.debian.org/cgit/collab-maint/supervisor.git/commit/?id=b3cde0be60078b02e5dc3942ad458f6af5e4dc84 (modulo the d/changelog change

Zach Welch (zwelch) wrote :

What needs to be done to get this fix released? This is a glaring bug for what is supposed to be an _LTS_ release.

Alex Gleason (alexgleason) wrote :

We just rebooted our web server (Django, Gunicorn) and encountered this problem. The website went offline and we are restoring from a backup. Please fix this. It is a critical issue, not medium.

Aditya (code-aditya) wrote :

@Alex and everyone affected by the bug. Peter has given a very nice detailed bug report with a known workaround.

Running the following once would ensure that supervisor always come up after a reboot:

    sudo systemctl enable supervisor

The above command is required to be run only once. So, get into a habit of running this command whenever you install Ubuntu 16.04 until the bug is fixed.

And, running the following command would bring supervisor up during the current boot:

    sudo systemctl start supervisor

Regards,
Aditya

Nish Aravamudan (nacc) on 2017-03-10
Changed in supervisor (Ubuntu):
status: Confirmed → Fix Released
Changed in supervisor (Ubuntu Yakkety):
status: New → Fix Released
Changed in supervisor (Ubuntu Xenial):
status: New → In Progress
assignee: nobody → Nish Aravamudan (nacc)
Nish Aravamudan (nacc) wrote :

I have uploaded a backport of the fix in Debian to:

https://launchpad.net/~nacc/+archive/ubuntu/lp1594740

as 3.2.0-2ubuntu0.1~ppa1.

Please test this version and report back. If it works, I will submit it for SRU in 16.04.

Peter Odding (peterodding) wrote :

Hi Nish and thank you so much for backporting a fix to xenial!

I can confirm that with the version of Supervisor in your PPA the original issue I reported is resolved, i.e. by the time 'apt install supervisor' returns, the Supervisor daemon has been started. Also both of the commands I mentioned in my opening post confirm the correct configuration:

peter@mbp> systemctl --quiet is-enabled supervisor && echo OK || echo ER
OK

peter@mbp> systemctl --quiet is-active supervisor && echo OK || echo ER
OK

Apart from this the Supervisor daemon seems to be working fine, as expected.

description: updated
Peter Odding (peterodding) wrote :

A minor follow up to my previous comment for anyone else testing this:

Correctly testing the fixed package is a bit subtle because if you simply run 'apt purge supervisor' the symbolic link /etc/systemd/system/multi-user.target.wants/supervisor.service (which was created by a previous manual invocation of 'systemctl enable supervisor') will remain and the next installation of the buggy 'supervisor' package will (confusingly enough) correctly start Supervisor after installation.

To summarize, I used the following commands to test this:

sudo apt purge supervisor
sudo rm /etc/systemd/system/multi-user.target.wants/supervisor.service
sudo apt install supervisor

# At this point, with a `clean re-installation' of the _buggy_
# package, the old and broken behavior that I originally reported can
# be reproduced. This was to verify that I could still reproduce the
# issue with up to date packages. It also confirmed that my manual
# `systemctl enable' invocation hadn't left any state behind on the
# filesystem.

sudo apt purge supervisor
sudo add-apt-repository ppa:nacc/lp1594740
sudo apt update
sudo apt install supervisor

# At this point the Supervisor daemon is correctly running straight
# after installation, without requiring `systemctl' commands from the
# operator.

Nish Aravamudan (nacc) on 2017-03-13
description: updated
Nish Aravamudan (nacc) wrote :

@peterodding: to double-check, if you are in the buggy state, and use the PPA version, does the service also start? Does it end up using the same systemd file as if you had used the version shipped in my PPA without installing the buggy version first?

Peter Odding (peterodding) wrote :

@nacc There are two ways to interpret your question :-) and both are valid inquiries with regards to regression potential, so I'll just check and answer both.

Just now I performed the following steps:

1. Purged the supervisor package,
2. removed the PPA `nacc/lp1594740',
3. ran `apt-get update',
4. ensured the symbolic link /etc/systemd/system/multi-user.target.wants/supervisor.service doesn't exist (indeed it doesn't, as expected),
5. reinstalled the supervisor package from Ubuntu's official repository (verified using `apt-cache policy supervisor'),
6. checked if the Supervisor daemon was started (it wasn't, because I just installed the buggy package),
7. reactivated the PPA `nacc/lp1594740',
8. ran `apt-get update && apt-get install supervisor', this upgraded the supervisor package to the version from the PPA.

After step 6 the Supervisor daemon is not running, but after step 8 it is running.

Theoretically there could be users out there who never noticed that Supervisor wasn't running after installation and after the upgrade to your fixed package Supervisor would `suddenly' be running for the first time on their system. But of course the whole point is for Supervisor to be running after installation! So this seems good to me.

Peter Odding (peterodding) wrote :

@nacc With regards to the other way to answer your question, I've now also tested the following sequence of events:

1. Purged the supervisor package,
2. removed the PPA `nacc/lp1594740',
3. ran `apt-get update',
4. ensured the symbolic link /etc/systemd/system/multi-user.target.wants/supervisor.service doesn't exist (indeed it doesn't, as expected),
5. reinstalled the supervisor package from Ubuntu's official repository (verified using `apt-cache policy supervisor'),
6. checked if the Supervisor daemon was started (it wasn't, because I just installed the buggy package),
7. ran `systemctl enable supervisor && systemctl start supervisor' as the documented workaround (this started and enabled the Supervisor daemon as expected),
7. reactivated the PPA `nacc/lp1594740',
8. ran `apt-get update && apt-get install supervisor', this upgraded the supervisor package to the version from the PPA.

After step 8 the Supervisor daemon has been restarted (its PID has changed, as I would expect from an upgrade to a package that contains a system daemon) but during the upgrade there were no conflicts, warnings or complaints about Supervisor having already been manually enabled previously. Again, this looks good to me!

Peter Odding (peterodding) wrote :

After carefully rereading @nacc's question I realized I missed an essential detail ("does it end up using the same systemd file"). I believe this is indeed the case, as intended (I assume?).

Here's what I get after a clean install of the fixed package:

peter@mbp> ls -ld /etc/systemd/system/multi-user.target.wants/supervisor.service
lrwxrwxrwx 1 root root 38 Mar 13 23:53 /etc/systemd/system/multi-user.target.wants/supervisor.service -> /lib/systemd/system/supervisor.service

Here's what I get after upgrading from the buggy version to the fixed version:

peter@mbp> ls -ld /etc/systemd/system/multi-user.target.wants/supervisor.service
lrwxrwxrwx 1 root root 38 Mar 13 23:55 /etc/systemd/system/multi-user.target.wants/supervisor.service -> /lib/systemd/system/supervisor.service

The target of the symlink is the same in both cases. The timestamps differ but that's just the creation time of the symlink AFAIK (from the moment 'systemctl enable' or an equivalent command is invoked).

Nish Aravamudan (nacc) wrote :

@Peter, thank you for the very thorough and helpful testing!

The primary point I meant to make with your last example was that the contents of the /lib/systemd/system/supervisor.service didn't change. But they shouldn't, because both buggy and fixed versions ship the same service file, just one doesn't setup the link properly.

Thanks!
-Nish

Peter Odding (peterodding) wrote :

@nacc:

After installation of the buggy package:

peter@mbp> sha1sum /lib/systemd/system/supervisor.service
88b0121e625b8ffe1cf6b0df3cf555bee8e7d3e9 /lib/systemd/system/supervisor.service

After applying the manual workarounds (just being thorough, of course I would expect 'systemctl' to just create the symlink and not touch anything in /lib):

peter@mbp> sha1sum /lib/systemd/system/supervisor.service
88b0121e625b8ffe1cf6b0df3cf555bee8e7d3e9 /lib/systemd/system/supervisor.service

After upgrading to the fixed package from the PPA:

peter@mbp> sha1sum /lib/systemd/system/supervisor.service
88b0121e625b8ffe1cf6b0df3cf555bee8e7d3e9 /lib/systemd/system/supervisor.service

Nish Aravamudan (nacc) wrote :

@Peter: excellent, thanks! Agreed on the expectations, I just wanted a double confirmation :)

Hello Peter, or anyone else affected,

Accepted supervisor into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/supervisor/3.2.0-2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in supervisor (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
Kyle Mulka (km-pg) wrote :

I just tested supervisor version 3.2.0-2ubuntu0.1 from xenial-proposed and it seems to work for me now. It starts as soon as it is installed and starts on boot. Thank you!

Peter Odding (peterodding) wrote :

Proving that the original issue still exists:

peter@mbp> sudo apt install supervisor
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  supervisor-doc
The following NEW packages will be installed:
  supervisor
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 252 kB of archives.
After this operation, 1,398 kB of additional disk space will be used.
Get:1 http://nl.archive.ubuntu.com/ubuntu xenial/universe amd64 supervisor all 3.2.0-2 [252 kB]
Fetched 252 kB in 0s (625 kB/s)
Selecting previously unselected package supervisor.
(Reading database ... 241488 files and directories currently installed.)
Preparing to unpack .../supervisor_3.2.0-2_all.deb ...
Unpacking supervisor (3.2.0-2) ...
Processing triggers for systemd (229-4ubuntu16) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up supervisor (3.2.0-2) ...
Processing triggers for systemd (229-4ubuntu16) ...
Processing triggers for ureadahead (0.100.0-19) ...
peter@mbp> ps auxw | grep '[s]upervisor'
peter@mbp>

No output, Supervisor is not running. Now I enable the xenial-proposed archive, opt-out of upgrading my main system from xenial-proposed and selectively install the fixed supervisor package from the proposed archive:

peter@mbp> sudo apt install supervisor/xenial-proposed
Reading package lists... Done
Building dependency tree
Reading state information... Done
Selected version '3.2.0-2ubuntu0.1' (Ubuntu:16.04/xenial-proposed [all]) for 'supervisor'
Suggested packages:
  supervisor-doc
The following packages will be upgraded:
  supervisor
1 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
Need to get 253 kB of archives.
After this operation, 1,024 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu xenial-proposed/universe amd64 supervisor all 3.2.0-2ubuntu0.1 [253 kB]
Fetched 253 kB in 1s (242 kB/s)
(Reading database ... 241609 files and directories currently installed.)
Preparing to unpack .../supervisor_3.2.0-2ubuntu0.1_all.deb ...
Unpacking supervisor (3.2.0-2ubuntu0.1) over (3.2.0-2) ...
Processing triggers for systemd (229-4ubuntu16) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up supervisor (3.2.0-2ubuntu0.1) ...
peter@mbp> ps auxw | grep '[s]upervisor'
root 15688 1.8 0.2 59148 18796 ? Ss 21:50 0:00 /usr/bin/python /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf

As you can see in the 'ps auxw' output the Supervisor daemon is now running.

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package supervisor - 3.2.0-2ubuntu0.1

---------------
supervisor (3.2.0-2ubuntu0.1) xenial; urgency=medium

  * Enable and start systemd service. Thanks to Orestis Ioannou
    <email address hidden>. (Closes LP: #1594740).

 -- Nishanth Aravamudan <email address hidden> Fri, 10 Mar 2017 10:43:23 -0800

Changed in supervisor (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for supervisor has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Peter Odding (peterodding) wrote :

Thanks to everyone involved in getting this fixed!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.