snmpd stop on host stops snmpd on LXD containers

[Arno van Huyssteen]

[Impact]

 * Stopping snmpd in Xenial can stop container processes of the same
   binary

 * Fix by matching to the PIDFILE that is already tracked by snmpd

[Test Case]

 * Set up a host (e.g. a VM being Xenial)
 * Set up at least one (or more) containers in it
 * Install and start snmpd in the Host
 * Install and start snmpd in the containers
 * you'll now see like:
    root@ubuntu-vm:/var/log# ps fxaw|grep snmp
    29382 ? S 0:00 \_ /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp
    29308 ? S 0:00 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp
  # One of these is from the container
 * now on the host stop your service, chances are you stop your guests instead
   root@ubuntu-vm:/var/log# service snmpd stop
   root@ubuntu-vm:/var/log# ps fxaw|grep snmp
   29426 pts/0 S+ 0:00 \_ grep --color=auto snmp
   # 29382 was from a lxd container in this case

[Regression Potential]

 * In case the pidfile stored by the snmpd daemon would be incorrect then
   the stop/restart actions would not match the process and fail to
   stop/restart. We didn't see that in our tests, but that is the
   regression I could think of.

[Other Info]

 * n/a

----

If you have ubuntu 16.04 containers running snmpd and you stop the snmpd on the LXD host system, it will also shut down all the snmpd instances on the containers (but will not restart them if you restart them on the LXD host) .. in fact, you even need to go back into the container, stop the snmpd before you can start it again.

See log below, viepovzat17 is the LXD host, viezmaaat10 is the ubuntu container:

  driver: lxc
  driver_version: 2.0.8
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.4.0-87-generic
  server: lxd
  server_pid: 4426
  server_version: "2.16"
  storage: zfs
  storage_version: 0.6.5.6-0ubuntu16

root@viezmaaat10:~# /etc/init.d/snmpd start
[ ok ] Starting snmpd (via systemctl): snmpd.service.
root@viezmaaat10:~# ps -eaf | grep snmp
snmp 1271 1 0 10:50 ? 00:00:00 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux mteTrigger mteTriggerConf -p /run/snmpd.pid
root 1291 757 0 10:50 ? 00:00:00 grep --color=auto snmp
root@viezmaaat10:~#
root@viezmaaat10:~# ps -eaf | grep snmp
snmp 1271 1 0 10:50 ? 00:00:00 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux mteTrigger mteTriggerConf -p /run/snmpd.pid
root 1293 757 0 10:50 ? 00:00:00 grep --color=auto snmp
root@viezmaaat10:~# exit

root@viepovzat17:~# ps -eaf | grep snmp
snmp 22757 1 0 10:45 ? 00:00:00 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux mteTrigger mteTriggerConf -p /run/snmpd.pid
100116 24118 5222 0 10:50 ? 00:00:00 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux mteTrigger mteTriggerConf -p /run/snmpd.pid
root 24153 21842 0 10:50 pts/1 00:00:00 grep --color=auto snmp
root@viepovzat17:~# /etc/init.d/snmpd stop
[ ok ] Stopping snmpd (via systemctl): snmpd.service.
root@viepovzat17:~# ps -eaf | grep snmp
root 24286 21842 0 10:50 pts/1 00:00:00 grep --color=auto snmp
root@viepovzat17:~#

[Stéphane Graber (stgraber)]

This is a bug in the snmpd init script, moving bug over there.

[Arno van Huyssteen (avanhuys)]

Has anyone had a chance to look into this bug in net-snmp yet?

[Andreas Hasenack (ahasenack)]

It's just using an executable match (--exec), instead of a pid:
  stop)
    log_daemon_msg "Stopping SNMP services:"
    start-stop-daemon --quiet --stop --oknodo --retry 3 --exec /usr/sbin/snmpd && [ ! -f $SNMP_PID ] || rm $SNMP_PID

That's the --exec bit.

[Christian Ehrhardt  (paelzer)]

Note: since 5.7.3+dfsg-1.4 a native systemd file is used which uses type=simple and MAINPID tracking.
So >= Artful this is already fixed.

Setting the status to fixed, but adding a Xenial task as there the issue still exists.

[Christian Ehrhardt  (paelzer)]

The old Xenial init file does actually already PID tracking via "-p $SNMP_PID" of snmpd itself.
That makes sure that the real daemon pid is in there and not other spawned tasks.

It has some cleanup to remove said file after stop (even if the daemon itself did not).
But in general it seems we can use that file as-is to use --pidfile

Interestingly, the reload action already does --pidfile "$SNMP_PID" --exec /usr/sbin/snmpd as I'd suggest.

A quick test modifying the script on a system worked, I'll propose the change for an SRU.

[Christian Ehrhardt  (paelzer)]

PPA to test at: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3166
MP: of the proposed change: https://code.launchpad.net/~paelzer/ubuntu/+source/net-snmp/+git/net-snmp/+merge/338019

[Christian Ehrhardt  (paelzer)]

- Ack on the MP (and some more tests)
- Updated the SRU template
- pushed the git ubuntu tag

Thereby uploaded the proposed change for the SRU team to evaluate.

[Chris J Arges (arges)]

Hello Arno, or anyone else affected,

Accepted net-snmp into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/net-snmp/5.7.3+dfsg-1ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Christian Ehrhardt  (paelzer)]

Xenial as-is
You can see on the +100000 UID the container is the first line:
$ ps axlf | grep snmpd | grep -v grep
5 100112 4553 2723 20 0 55228 5952 - S ? 0:00 \_ /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux mteTrigger mteTriggerConf -p /run/snmpd.pid
5 112 3398 1 20 0 63340 7672 - S ? 0:00 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux mteTrigger mteTriggerConf -p /run/snmpd.pid

$ sudo systemctl restart snmpd
$ ps axlf | grep snmpd | grep -v grep
5 112 4656 1 20 0 63344 7820 - S ? 0:00 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux mteTrigger mteTriggerConf -p /run/snmpd.pid

Containers view:
Feb 21 16:06:35 x1 systemd[1]: Started LSB: SNMP agents.
Feb 21 16:06:35 x1 snmpd[1208]: NET-SNMP version 5.7.3
Feb 21 16:08:03 x1 snmpd[1208]: Received TERM or STOP signal... shutting down...

Upgrade to snmpd from proposed:
$ sudo apt install snmpd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libsnmp30
Suggested packages:
  snmptrapd
The following packages will be upgraded:
  libsnmp30 snmpd
2 upgraded, 0 newly installed, 0 to remove and 39 not upgraded.
Need to get 868 kB of archives.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 snmpd amd64 5.7.3+dfsg-1ubuntu4.1 [57,2 kB]
Get:2 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 libsnmp30 amd64 5.7.3+dfsg-1ubuntu4.1 [811 kB]
Fetched 868 kB in 0s (2.779 kB/s)
Preconfiguring packages ...
(Reading database ... 54148 files and directories currently installed.)
Preparing to unpack .../snmpd_5.7.3+dfsg-1ubuntu4.1_amd64.deb ...
Unpacking snmpd (5.7.3+dfsg-1ubuntu4.1) over (5.7.3+dfsg-1ubuntu4) ...
Preparing to unpack .../libsnmp30_5.7.3+dfsg-1ubuntu4.1_amd64.deb ...
Unpacking libsnmp30:amd64 (5.7.3+dfsg-1ubuntu4.1) over (5.7.3+dfsg-1ubuntu4) ...
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Setting up libsnmp30:amd64 (5.7.3+dfsg-1ubuntu4.1) ...
Setting up snmpd (5.7.3+dfsg-1ubuntu4.1) ...

$ ps axlf | grep snmpd | grep -v grep
5 100112 5601 2723 20 0 55228 5668 - S ? 0:00 \_ /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux mteTrigger mteTriggerConf -p /run/snmpd.pid
5 112 5573 1 20 0 63344 7684 - S ? 0:00 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux mteTrigger mteTriggerConf -p /run/snmpd.pid

Host:
$ sudo systemctl restart snmpd
Container still:
Active: active (running) since Wed 2018-02-21 16:10:39 UTC; 1min 3s ago

$ ps axlf | grep snmpd | grep -v grep
5 100112 5601 2723 20 0 55228 5668 - S ? 0:00 \_ /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snmp -I -smux mteTrigger mteTriggerConf -p /run/snmpd.pid
5 112 5660 1 20 0 63340 7656 - S ? 0:00 /usr/sbi...

[Launchpad Janitor (janitor)]

This bug was fixed in the package net-snmp - 5.7.3+dfsg-1ubuntu4.1

---------------
net-snmp (5.7.3+dfsg-1ubuntu4.1) xenial; urgency=medium

  * d/snmpd.init: also match start-stop-daemon against pidfile to avoid
    killing extra snmpd processes for example in container (LP: #1720109)

 -- Christian Ehrhardt <email address hidden> Tue, 20 Feb 2018 14:53:51 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for net-snmp has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.