Upgrade of erlang-base left rabbitmq-server dead

[Nils Toedtmann]

Last night, unattended-upgrades upgraded the erlang packages on our RabbitMQ server from 1:18.3-dfsg-1ubuntu3 to 1:18.3-dfsg-1ubuntu3.1. erlang-base's prerm script successfully found and stopped rabbitmq-server, and the postinst script tried to start it again:

# From /var/log/apt/term.log, same in /var/log/unattended-upgrades/unattended-upgrades-dpkg.log
#...
Preparing to unpack .../erlang-base_1%3a18.3-dfsg-1ubuntu3.1_amd64.deb ...
Searching for services which depend on erlang and should be stopped...found: rabbitmq-server.
Stopping services which depend on erlang
  rabbitmq-server: stopping...done.
Services stopped successfully.
Killing epmd...it is not running.
Unpacking erlang-base (1:18.3-dfsg-1ubuntu3.1) over (1:18.3-dfsg-1ubuntu3) ...
Preparing to unpack .../erlang-nox_1%3a18.3-dfsg-1ubuntu3.1_all.deb ...
Unpacking erlang-nox (1:18.3-dfsg-1ubuntu3.1) over (1:18.3-dfsg-1ubuntu3) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up erlang-base (1:18.3-dfsg-1ubuntu3.1) ...
Searching for services which depend on erlang and should be started...found: rabbitmq-server.
Starting services which depend on erlang
  rabbitmq-server: starting...done.
Services started successfully.
#...

I looked into erlang-base.prerm and erlang-base.postinst and ran the relevant snippets manually, and it looks like they would have used "invoke-rc.d":

   invoke-rc.d rabbitmq-server stop > /dev/null 2>&1 # from erlang-base.prerm
   invoke-rc.d rabbitmq-server start > /dev/null 2>&1 # from erlang-base.postinst

But while the stop command succeeded, the start command did not, and there isn't any evidence in the logs that it was ever issued. This is the reason why I think this is a bug in the erlang package (or possibly invoke-rc.d/init-system-helpers?), and not in rabbitmq-server:

# journalctl
Feb 16 06:30:03 rabbit-b systemd[1]: Starting Daily apt upgrade and clean activities...
Feb 16 06:30:48 rabbit-b systemd[1]: Reloading.
Feb 16 06:30:49 rabbit-b systemd[1]: Started ACPI event daemon.
Feb 16 06:30:49 rabbit-b systemd[1]: Stopping RabbitMQ Messaging Server...
Feb 16 06:30:50 rabbit-b rabbitmq[6596]: Stopping and halting node 'rabbit@rabbit-b' ...
Feb 16 06:30:51 rabbit-b systemd[1]: Stopped RabbitMQ Messaging Server.
Feb 16 06:30:52 rabbit-b systemd[1]: Reloading.
Feb 16 06:30:52 rabbit-b systemd[1]: Started ACPI event daemon.
Feb 16 06:30:54 rabbit-b systemd[1]: Reloading.
Feb 16 06:30:54 rabbit-b systemd[1]: Started ACPI event daemon.
# ... and the next log entry is from after unattended-upgrades had ended at 06:30:56

# /<email address hidden>
=INFO REPORT==== 16-Feb-2018::06:30:50 ===
Stopping RabbitMQ
#... lots of AMQP connection closures
=INFO REPORT==== 16-Feb-2018::06:30:51 ===
Halting Erlang VM
# ... and the next log entry is from 2h later when we manually started RabbitMQ again

This is Ubuntu 16.04.2 LTS with all security patches and the latest rabbitmq-server (3.5.7-1ubuntu0.16.04.2)

PS: Where are the sources for erlang-base.prerm and erlang-base.postinst? https://packages.debian.org/source/stretch/erlang points to http://anonscm.debian.org/viewvc/pkg-erlang/erlang/trunk/, but that link is dead

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.