| 2018-08-01 01:21:39 |
Ivan Kurnosov |
bug |
|
|
added bug |
| 2018-08-02 12:32:47 |
Robie Basak |
rabbitmq-server (Ubuntu): status |
New |
Incomplete |
|
| 2018-10-02 04:17:33 |
Launchpad Janitor |
rabbitmq-server (Ubuntu): status |
Incomplete |
Expired |
|
| 2018-10-13 16:34:32 |
Alex Tomkins |
attachment added |
|
rabbitmq-stop-shutdown.patch https://bugs.launchpad.net/ubuntu/+source/rabbitmq-server/+bug/1784757/+attachment/5200657/+files/rabbitmq-stop-shutdown.patch |
|
| 2018-10-13 16:36:50 |
Alex Tomkins |
rabbitmq-server (Ubuntu): status |
Expired |
Confirmed |
|
| 2018-10-13 16:48:23 |
Alex Tomkins |
bug |
|
|
added subscriber Alex Tomkins |
| 2018-10-13 20:19:18 |
Ubuntu Foundations Team Bug Bot |
tags |
|
patch |
|
| 2018-10-13 20:19:26 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Review Team |
| 2018-10-15 14:13:50 |
Christian Ehrhardt |
bug |
|
|
added subscriber Ubuntu Server |
| 2018-10-15 14:13:55 |
Christian Ehrhardt |
tags |
patch |
patch server-next |
|
| 2018-10-15 14:18:45 |
Christian Ehrhardt |
rabbitmq-server (Ubuntu): importance |
Undecided |
Medium |
|
| 2018-12-12 11:49:30 |
Kirill Goncharov |
bug |
|
|
added subscriber Kirill Goncharov |
| 2019-05-10 07:22:17 |
Christian Ehrhardt |
rabbitmq-server (Ubuntu): status |
Confirmed |
Incomplete |
|
| 2019-06-07 23:43:24 |
Bryce Harrington |
tags |
patch server-next |
|
|
| 2019-06-09 20:37:58 |
Bryce Harrington |
tags |
|
server-next |
|
| 2019-06-09 20:38:03 |
Bryce Harrington |
rabbitmq-server (Ubuntu): status |
Incomplete |
Triaged |
|
| 2019-06-09 20:44:21 |
Bryce Harrington |
nominated for series |
|
Ubuntu Cosmic |
|
| 2019-06-09 20:44:21 |
Bryce Harrington |
bug task added |
|
rabbitmq-server (Ubuntu Cosmic) |
|
| 2019-06-09 20:44:21 |
Bryce Harrington |
nominated for series |
|
Ubuntu Bionic |
|
| 2019-06-09 20:44:21 |
Bryce Harrington |
bug task added |
|
rabbitmq-server (Ubuntu Bionic) |
|
| 2019-06-09 20:46:48 |
Bryce Harrington |
description |
When I run `systemctl restart rabbitmq-server` it waits for 90 seconds then systemd sends SIGKILL to it.
Presumably the `epmd` process does not receive SIGTERM, since if I run `kill 1493` (or whatever pid it currently is) then restart happens straight after that successfully
● rabbitmq-server.service - RabbitMQ Messaging Server
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: deactivating (final-sigterm) since Wed 2018-08-01 01:17:04 UTC; 7s ago
Process: 1183 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
Process: 178 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=0/SUCCESS)
Process: 177 ExecStart=/usr/sbin/rabbitmq-server (code=killed, signal=TERM)
Main PID: 177 (code=killed, signal=TERM)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/rabbitmq-server.service
└─1493 /usr/lib/erlang/erts-9.2/bin/epmd -daemon
Aug 01 01:11:20 rmq-1 systemd[1]: rabbitmq-server.service: Failed to reset devices.list: Operation not permitted
Aug 01 01:11:20 rmq-1 systemd[1]: Starting RabbitMQ Messaging Server...
Aug 01 01:11:25 rmq-1 rabbitmq[178]: Waiting for 'rabbit@rmq-1'
Aug 01 01:11:25 rmq-1 rabbitmq[178]: pid is 204
Aug 01 01:11:30 rmq-1 systemd[1]: Started RabbitMQ Messaging Server.
Aug 01 01:17:04 rmq-1 systemd[1]: Stopping RabbitMQ Messaging Server...
Aug 01 01:17:06 rmq-1 rabbitmq[1183]: Stopping and halting node 'rabbit@rmq-1' |
[Impact]
TBD
[Test Case]
TBD
[Regression Potential]
TBD
[Fix]
TBD
The fix is available upstream in Debian as of 3.7.6, so would need backported for bionic and cosmic:
rabbitmq-server | 3.5.7-1 | xenial | source, all
rabbitmq-server | 3.6.10-1 | bionic | source, all
rabbitmq-server | 3.6.10-1 | cosmic | source, all
rabbitmq-server | 3.7.8-4ubuntu2 | disco | source, all
rabbitmq-server | 3.7.8-4ubuntu2 | eoan | source, all
[Discussion]
TBD
[Original Report]
When I run `systemctl restart rabbitmq-server` it waits for 90 seconds then systemd sends SIGKILL to it.
Presumably the `epmd` process does not receive SIGTERM, since if I run `kill 1493` (or whatever pid it currently is) then restart happens straight after that successfully
● rabbitmq-server.service - RabbitMQ Messaging Server
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: deactivating (final-sigterm) since Wed 2018-08-01 01:17:04 UTC; 7s ago
Process: 1183 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
Process: 178 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=0/SUCCESS)
Process: 177 ExecStart=/usr/sbin/rabbitmq-server (code=killed, signal=TERM)
Main PID: 177 (code=killed, signal=TERM)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/rabbitmq-server.service
└─1493 /usr/lib/erlang/erts-9.2/bin/epmd -daemon
Aug 01 01:11:20 rmq-1 systemd[1]: rabbitmq-server.service: Failed to reset devices.list: Operation not permitted
Aug 01 01:11:20 rmq-1 systemd[1]: Starting RabbitMQ Messaging Server...
Aug 01 01:11:25 rmq-1 rabbitmq[178]: Waiting for 'rabbit@rmq-1'
Aug 01 01:11:25 rmq-1 rabbitmq[178]: pid is 204
Aug 01 01:11:30 rmq-1 systemd[1]: Started RabbitMQ Messaging Server.
Aug 01 01:17:04 rmq-1 systemd[1]: Stopping RabbitMQ Messaging Server...
Aug 01 01:17:06 rmq-1 rabbitmq[1183]: Stopping and halting node 'rabbit@rmq-1' |
|
| 2019-06-09 20:46:53 |
Bryce Harrington |
rabbitmq-server (Ubuntu Bionic): status |
New |
Triaged |
|
| 2019-06-09 20:46:55 |
Bryce Harrington |
rabbitmq-server (Ubuntu Cosmic): status |
New |
Triaged |
|
| 2019-06-09 20:46:57 |
Bryce Harrington |
rabbitmq-server (Ubuntu Cosmic): importance |
Undecided |
Medium |
|
| 2019-06-09 20:46:59 |
Bryce Harrington |
rabbitmq-server (Ubuntu Bionic): importance |
Undecided |
Medium |
|
| 2019-06-09 20:47:01 |
Bryce Harrington |
rabbitmq-server (Ubuntu): status |
Triaged |
Fix Released |
|
| 2019-06-09 20:47:17 |
Bryce Harrington |
summary |
rabbitmq-server does not properly shutdown |
[SRU] rabbitmq-server does not properly shutdown |
|
| 2019-07-06 00:08:07 |
Bryce Harrington |
bug watch added |
|
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=812342 |
|
| 2019-07-06 00:08:07 |
Bryce Harrington |
bug task added |
|
rabbitmq-server (Debian) |
|
| 2019-07-06 02:03:37 |
Bug Watch Updater |
rabbitmq-server (Debian): status |
Unknown |
Fix Released |
|
| 2019-07-09 01:35:34 |
Bryce Harrington |
rabbitmq-server (Ubuntu Bionic): status |
Triaged |
Incomplete |
|
| 2020-07-02 23:05:28 |
Sergio Durigan Junior |
rabbitmq-server (Ubuntu Cosmic): status |
Triaged |
Won't Fix |
|
| 2020-07-02 23:18:07 |
Sergio Durigan Junior |
bug |
|
|
added subscriber Sergio Durigan Junior |
| 2020-07-03 16:14:08 |
Sergio Durigan Junior |
rabbitmq-server (Ubuntu Bionic): status |
Incomplete |
Triaged |
|
| 2020-07-06 19:15:59 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~sergiodj/ubuntu/+source/rabbitmq-server/+git/rabbitmq-server/+merge/386905 |
|
| 2020-07-07 12:31:00 |
Dan Streetman |
bug |
|
|
added subscriber Nicolas Bock |
| 2020-07-07 12:31:08 |
Dan Streetman |
bug |
|
|
added subscriber Matthew Ruffell |
| 2020-07-10 01:26:39 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~sergiodj/ubuntu/+source/rabbitmq-server/+git/rabbitmq-server/+merge/387177 |
|
| 2020-07-13 20:34:58 |
Sergio Durigan Junior |
rabbitmq-server (Ubuntu Bionic): assignee |
|
Sergio Durigan Junior (sergiodj) |
|
| 2020-07-13 20:54:17 |
Sergio Durigan Junior |
description |
[Impact]
TBD
[Test Case]
TBD
[Regression Potential]
TBD
[Fix]
TBD
The fix is available upstream in Debian as of 3.7.6, so would need backported for bionic and cosmic:
rabbitmq-server | 3.5.7-1 | xenial | source, all
rabbitmq-server | 3.6.10-1 | bionic | source, all
rabbitmq-server | 3.6.10-1 | cosmic | source, all
rabbitmq-server | 3.7.8-4ubuntu2 | disco | source, all
rabbitmq-server | 3.7.8-4ubuntu2 | eoan | source, all
[Discussion]
TBD
[Original Report]
When I run `systemctl restart rabbitmq-server` it waits for 90 seconds then systemd sends SIGKILL to it.
Presumably the `epmd` process does not receive SIGTERM, since if I run `kill 1493` (or whatever pid it currently is) then restart happens straight after that successfully
● rabbitmq-server.service - RabbitMQ Messaging Server
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: deactivating (final-sigterm) since Wed 2018-08-01 01:17:04 UTC; 7s ago
Process: 1183 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
Process: 178 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=0/SUCCESS)
Process: 177 ExecStart=/usr/sbin/rabbitmq-server (code=killed, signal=TERM)
Main PID: 177 (code=killed, signal=TERM)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/rabbitmq-server.service
└─1493 /usr/lib/erlang/erts-9.2/bin/epmd -daemon
Aug 01 01:11:20 rmq-1 systemd[1]: rabbitmq-server.service: Failed to reset devices.list: Operation not permitted
Aug 01 01:11:20 rmq-1 systemd[1]: Starting RabbitMQ Messaging Server...
Aug 01 01:11:25 rmq-1 rabbitmq[178]: Waiting for 'rabbit@rmq-1'
Aug 01 01:11:25 rmq-1 rabbitmq[178]: pid is 204
Aug 01 01:11:30 rmq-1 systemd[1]: Started RabbitMQ Messaging Server.
Aug 01 01:17:04 rmq-1 systemd[1]: Stopping RabbitMQ Messaging Server...
Aug 01 01:17:06 rmq-1 rabbitmq[1183]: Stopping and halting node 'rabbit@rmq-1' |
[Impact]
The systemd file rabbitmq-server.service on Bionic uses "Type=simple" when defining the service, but unfortunately this doesn't work very well for rabbitmq-server. In certain situations, systemd will fail to keep track of a start/stop/restart event, and will hang for 90 seconds before giving the prompt back to the user. Another problem is that rabbitmq-server must start after the epmd service, so we need to explicitly declare this dependency in the service file.
[Test Case]
Although I was able to reproduce this almost 100% of the time, there were rare occasions when the restart procedure finished normally. I was also only able to reproduce it using a bionic VM, not a container. If you have multipass or lxd configured to launch VMs, that should be easy.
The steps are:
$ lxc launch ubuntu-daily:bionic --vm bug1784757-rabbitmq-server # or use multipass
$ lxc shell bug1784757-rabbitmq-server
# apt update
# apt install rabbitmq-server -y
# systemctl restart rabbitmq-server.service
In a normal scenario, the restart should take around 3 seconds or less. With the bug, it takes around 90 seconds. If you can't reproduce it, try running "systemctl restart" again. A quick way to trigger it is to run a for loop like:
# for i in $(seq 10); do time systemctl restart rabbitmq-server.service ; done
[Regression Potential]
* Because rabbitmq-server implements systemd's "Type=notify" using socat to communicate with systemd-notify over a socket, we will be introducing another point of failure (socat) in the mix.
* So, albeit unlikely, there may be a problem when using socat. That would not be a regression, though, since the outcome would be the same as we have today: "systemctl restart" would not properly work, even though the service did restart.
[Original Report]
When I run `systemctl restart rabbitmq-server` it waits for 90 seconds then systemd sends SIGKILL to it.
Presumably the `epmd` process does not receive SIGTERM, since if I run `kill 1493` (or whatever pid it currently is) then restart happens straight after that successfully
● rabbitmq-server.service - RabbitMQ Messaging Server
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: deactivating (final-sigterm) since Wed 2018-08-01 01:17:04 UTC; 7s ago
Process: 1183 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
Process: 178 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=0/SUCCESS)
Process: 177 ExecStart=/usr/sbin/rabbitmq-server (code=killed, signal=TERM)
Main PID: 177 (code=killed, signal=TERM)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/rabbitmq-server.service
└─1493 /usr/lib/erlang/erts-9.2/bin/epmd -daemon
Aug 01 01:11:20 rmq-1 systemd[1]: rabbitmq-server.service: Failed to reset devices.list: Operation not permitted
Aug 01 01:11:20 rmq-1 systemd[1]: Starting RabbitMQ Messaging Server...
Aug 01 01:11:25 rmq-1 rabbitmq[178]: Waiting for 'rabbit@rmq-1'
Aug 01 01:11:25 rmq-1 rabbitmq[178]: pid is 204
Aug 01 01:11:30 rmq-1 systemd[1]: Started RabbitMQ Messaging Server.
Aug 01 01:17:04 rmq-1 systemd[1]: Stopping RabbitMQ Messaging Server...
Aug 01 01:17:06 rmq-1 rabbitmq[1183]: Stopping and halting node 'rabbit@rmq-1' |
|
| 2020-07-28 20:32:36 |
Brian Murray |
rabbitmq-server (Ubuntu Bionic): status |
Triaged |
Fix Committed |
|
| 2020-07-28 20:32:38 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2020-07-28 20:32:40 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2020-07-28 20:32:45 |
Brian Murray |
tags |
server-next |
server-next verification-needed verification-needed-bionic |
|
| 2020-10-28 19:29:30 |
Brian Murray |
tags |
server-next verification-needed verification-needed-bionic |
removal-candidate server-next verification-needed verification-needed-bionic |
|
| 2020-10-29 19:26:16 |
Sergio Durigan Junior |
tags |
removal-candidate server-next verification-needed verification-needed-bionic |
removal-candidate server-next verification-done-bionic verification-needed |
|
| 2020-10-29 23:42:27 |
Launchpad Janitor |
rabbitmq-server (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2020-10-29 23:42:30 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
