health-manager subprocess remain alive after restart the service

[qi zhang]

[root@controller1 ~]# systemctl status octavia-health-manager.service
● octavia-health-manager.service - OpenStack Octavia Health-Manager service
   Loaded: loaded (/usr/lib/systemd/system/octavia-health-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-03-20 14:31:48 CST; 19h ago
 Main PID: 1698 (python)
   CGroup: /system.slice/octavia-health-manager.service
           ├─ 1157 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file...
           ├─ 1698 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file...
           ├─ 1720 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file...
           ├─ 7347 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file...
           ├─30731 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file...
           └─32732 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file...

Mar 20 14:31:50 controller1 python[1698]: self._target(*self._args, **self._kwargs)
Mar 20 14:31:50 controller1 python[1698]: File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/octavia/cmd/health_ma...listener
Mar 20 14:31:50 controller1 python[1698]: update_db.UpdateStatsDb())
Mar 20 14:31:50 controller1 python[1698]: File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/octavia/amphorae/driv...__init__
Mar 20 14:31:50 controller1 python[1698]: self.update(self.key, self.ip, self.port)
Mar 20 14:31:50 controller1 python[1698]: File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/octavia/amphorae/driv...n update
Mar 20 14:31:50 controller1 python[1698]: self.sock.bind(self.sockaddr)
Mar 20 14:31:50 controller1 python[1698]: File "/usr/lib64/python2.7/socket.py", line 224, in meth
Mar 20 14:31:50 controller1 python[1698]: return getattr(self._sock,name)(*args)
Mar 20 14:31:50 controller1 python[1698]: error: [Errno 98] Address already in use
Hint: Some lines were ellipsized, use -l to show in full.

Exactly octavia-health-manager will only create two subprocess. However there are 6 process of octavia-health-manager,this is not correct. And there is a error :[Errno 98] Address already in use. Obviously the reason is that
hm_listener_proc subprocess is still alive after restart the octavia-health-manager service. It still listen to port 5555.
[root@controller1 ~]# netstat -nulp |grep 5555
udp 0 0 192.168.26.131:5555 0.0.0.0:* 7347/python
[root@controller1 ~]# ps -fp 7347
UID PID PPID C STIME TTY TIME CMD
octavia 7347 1 0 Mar17 ? 00:10:13 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log

[root@controller1 ~]# ps -fp 7347
UID PID PPID C STIME TTY TIME CMD
octavia 7347 1 0 Mar17 ? 00:10:13 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log

[root@controller1 ~]# ps -ef |grep octavia-health
octavia 1157 1 0 Mar20 ? 00:01:26 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log
octavia 1698 1 0 Mar20 ? 00:00:00 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log
octavia 1720 1698 0 Mar20 ? 00:01:25 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log
octavia 7347 1 0 Mar17 ? 00:10:14 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log
root 27477 26550 0 09:57 pts/10 00:00:00 grep --color=auto octavia-health
octavia 30731 1 0 Mar20 ? 00:01:45 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log
octavia 32732 1 0 Mar20 ? 00:01:28 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log

Take care of STIME column , it is weird that process 7347 STIME is Mar 17!

Here is /usr/lib/systemd/system/octavia-health-manager.service
[Unit]
Description=OpenStack Octavia Health-Manager service
After=syslog.target network.target

[Service]
Type=simple
User=octavia
ExecStart=/opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log
PrivateTmp=false
KillMode=process

[Install]
WantedBy=multi-user.target

[Michael Johnson (johnsom)]

Hello,

I think this is working as designed. The subprocesses will not shutdown until they finish processing the currently active processing. For example, if the health manager is in the process of failing over a failed amphroa, it will not exit until the failover is complete or has gone into error.

Can you post the fill health manager logs? Also, please state which version of octavia you are running.

[Michael Johnson (johnsom)]

FYI, we no longer use launchpad, all bugs have been migrated to Storyboard.
This one in particular was previously resolved and closed.

[Gregory Thiemonge (gthiemonge)]

Abandoned after re-enabling the Octavia launchpad.