Activity log for bug #1674522

Date Who What changed Old value New value Message
2017-03-21 02:00:22 qi zhang bug added bug
2017-03-21 02:01:04 qi zhang octavia: assignee qi zhang (saiisgod)
2017-03-21 02:13:18 Michael Johnson octavia: status New Incomplete
2017-03-21 02:16:00 qi zhang description [root@controller1 ~]# systemctl status octavia-health-manager.service ● octavia-health-manager.service - OpenStack Octavia Health-Manager service Loaded: loaded (/usr/lib/systemd/system/octavia-health-manager.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2017-03-20 14:31:48 CST; 19h ago Main PID: 1698 (python) CGroup: /system.slice/octavia-health-manager.service ├─ 1157 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file... ├─ 1698 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file... ├─ 1720 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file... ├─ 7347 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file... ├─30731 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file... └─32732 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file... Mar 20 14:31:50 controller1 python[1698]: self._target(*self._args, **self._kwargs) Mar 20 14:31:50 controller1 python[1698]: File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/octavia/cmd/health_ma...listener Mar 20 14:31:50 controller1 python[1698]: update_db.UpdateStatsDb()) Mar 20 14:31:50 controller1 python[1698]: File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/octavia/amphorae/driv...__init__ Mar 20 14:31:50 controller1 python[1698]: self.update(self.key, self.ip, self.port) Mar 20 14:31:50 controller1 python[1698]: File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/octavia/amphorae/driv...n update Mar 20 14:31:50 controller1 python[1698]: self.sock.bind(self.sockaddr) Mar 20 14:31:50 controller1 python[1698]: File "/usr/lib64/python2.7/socket.py", line 224, in meth Mar 20 14:31:50 controller1 python[1698]: return getattr(self._sock,name)(*args) Mar 20 14:31:50 controller1 python[1698]: error: [Errno 98] Address already in use Hint: Some lines were ellipsized, use -l to show in full. Exactly octavia-health-manager will only create two subprocess. However there are 6 process of octavia-health-manager,this is not correct. And there is a error :[Errno 98] Address already in use. Obviously the reason is that hm_listener_proc subprocess is still alive after restart the octavia-health-manager service. It still listen to port 5555. [root@controller1 ~]# netstat -nulp |grep 5555 udp 0 0 192.168.26.131:5555 0.0.0.0:* 7347/python [root@controller1 ~]# ps -fp 7347 UID PID PPID C STIME TTY TIME CMD octavia 7347 1 0 Mar17 ? 00:10:13 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log [root@controller1 ~]# ps -fp 7347 UID PID PPID C STIME TTY TIME CMD octavia 7347 1 0 Mar17 ? 00:10:13 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log [root@controller1 ~]# ps -ef |grep octavia-health octavia 1157 1 0 Mar20 ? 00:01:26 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log octavia 1698 1 0 Mar20 ? 00:00:00 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log octavia 1720 1698 0 Mar20 ? 00:01:25 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log octavia 7347 1 0 Mar17 ? 00:10:14 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log root 27477 26550 0 09:57 pts/10 00:00:00 grep --color=auto octavia-health octavia 30731 1 0 Mar20 ? 00:01:45 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log octavia 32732 1 0 Mar20 ? 00:01:28 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log Take care of STIME column , it is weird that process 7347 STIME is Mar 17! [root@controller1 ~]# systemctl status octavia-health-manager.service ● octavia-health-manager.service - OpenStack Octavia Health-Manager service    Loaded: loaded (/usr/lib/systemd/system/octavia-health-manager.service; enabled; vendor preset: disabled)    Active: active (running) since Mon 2017-03-20 14:31:48 CST; 19h ago  Main PID: 1698 (python)    CGroup: /system.slice/octavia-health-manager.service            ├─ 1157 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file...            ├─ 1698 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file...            ├─ 1720 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file...            ├─ 7347 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file...            ├─30731 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file...            └─32732 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file... Mar 20 14:31:50 controller1 python[1698]: self._target(*self._args, **self._kwargs) Mar 20 14:31:50 controller1 python[1698]: File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/octavia/cmd/health_ma...listener Mar 20 14:31:50 controller1 python[1698]: update_db.UpdateStatsDb()) Mar 20 14:31:50 controller1 python[1698]: File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/octavia/amphorae/driv...__init__ Mar 20 14:31:50 controller1 python[1698]: self.update(self.key, self.ip, self.port) Mar 20 14:31:50 controller1 python[1698]: File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/octavia/amphorae/driv...n update Mar 20 14:31:50 controller1 python[1698]: self.sock.bind(self.sockaddr) Mar 20 14:31:50 controller1 python[1698]: File "/usr/lib64/python2.7/socket.py", line 224, in meth Mar 20 14:31:50 controller1 python[1698]: return getattr(self._sock,name)(*args) Mar 20 14:31:50 controller1 python[1698]: error: [Errno 98] Address already in use Hint: Some lines were ellipsized, use -l to show in full. Exactly octavia-health-manager will only create two subprocess. However there are 6 process of octavia-health-manager,this is not correct. And there is a error :[Errno 98] Address already in use. Obviously the reason is that hm_listener_proc subprocess is still alive after restart the octavia-health-manager service. It still listen to port 5555. [root@controller1 ~]# netstat -nulp |grep 5555 udp 0 0 192.168.26.131:5555 0.0.0.0:* 7347/python [root@controller1 ~]# ps -fp 7347 UID PID PPID C STIME TTY TIME CMD octavia 7347 1 0 Mar17 ? 00:10:13 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log [root@controller1 ~]# ps -fp 7347 UID PID PPID C STIME TTY TIME CMD octavia 7347 1 0 Mar17 ? 00:10:13 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log [root@controller1 ~]# ps -ef |grep octavia-health octavia 1157 1 0 Mar20 ? 00:01:26 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log octavia 1698 1 0 Mar20 ? 00:00:00 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log octavia 1720 1698 0 Mar20 ? 00:01:25 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log octavia 7347 1 0 Mar17 ? 00:10:14 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log root 27477 26550 0 09:57 pts/10 00:00:00 grep --color=auto octavia-health octavia 30731 1 0 Mar20 ? 00:01:45 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log octavia 32732 1 0 Mar20 ? 00:01:28 /opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log Take care of STIME column , it is weird that process 7347 STIME is Mar 17! Here is /usr/lib/systemd/system/octavia-health-manager.service [Unit] Description=OpenStack Octavia Health-Manager service After=syslog.target network.target [Service] Type=simple User=octavia ExecStart=/opt/openstack/octavia/octavia-env/bin/python /opt/openstack/octavia/octavia-env/bin/octavia-health-manager --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/health-manager.log PrivateTmp=false KillMode=process [Install] WantedBy=multi-user.target
2017-03-21 02:17:08 qi zhang information type Public Public Security
2017-04-20 18:27:07 Michael Johnson information type Public Security Public
2023-03-31 08:15:09 Gregory Thiemonge octavia: status Incomplete Invalid
2023-03-31 08:15:09 Gregory Thiemonge octavia: assignee qi zhang (saiisgod)
2023-03-31 08:15:12 Gregory Thiemonge tags auto-abandon