virtualbmc processes silently die over time

[Steven Hardy]

I've been seeing this for a while now, so adding a bug where we can hopefully capture more evidence to figure out the root cause.

My quickstart deployed VM development environment seems to gradually lose vbmc processes over time, so every couple of days I need to restart the service, e.g:

(undercloud) [stack@undercloud ~]$ ps ax | grep bmc
 2321 ? Sl 4:33 /usr/bin/python2 /usr/bin/vbmc start compute_0
 2522 ? Sl 4:32 /usr/bin/python2 /usr/bin/vbmc start compute_2
18168 pts/0 R+ 0:00 grep --color=auto bmc
(undercloud) [stack@undercloud ~]$ sudo systemctl restart virtualbmc
(undercloud) [stack@undercloud ~]$ ps ax | grep bmc
18223 ? Sl 0:00 /usr/bin/python2 /usr/bin/vbmc start compute_0
18233 ? Sl 0:00 /usr/bin/python2 /usr/bin/vbmc start compute_1
18241 ? Sl 0:00 /usr/bin/python2 /usr/bin/vbmc start compute_2
18250 ? Sl 0:00 /usr/bin/python2 /usr/bin/vbmc start control_0
18259 ? Sl 0:00 /usr/bin/python2 /usr/bin/vbmc start control_1
18279 ? Sl 0:00 /usr/bin/python2 /usr/bin/vbmc start control_2
18282 pts/0 S+ 0:00 grep --color=auto bmc

The system has plenty of memory and swap, so I don't think it's due to the OOM killer, and there's nothing in the logs AFAICS:

(undercloud) [stack@undercloud ~]$ free -m
              total used free shared buff/cache available
Mem: 12015 7661 2241 0 2112 3933
Swap: 4095 1187 2908

(undercloud) [stack@undercloud ~]$ sudo cat /var/log/virtualbmc/virtualbmc.log | grep -i error
(undercloud) [stack@undercloud ~]$

[Steven Hardy (shardy)]

I haven't seen this in a while so will close and reopen if it happens again

[Launchpad Janitor (janitor)]

[Expired for tripleo because there has been no activity for 60 days.]