OVS upgrade suspected to be causing OVS startup failures on containers

[Paul Goins]

Hi,

On an O7k cloud, we have Octavia deployed in LXD containers, and an openvswitch app deployed as subordinates to those units.

I recently had to upgrade all packages per customer request and, in turn, restart the nodes. Upon restarting, I found that ovs-vswitchd.service fails to start on the upgraded Octavia unit.

I've been applying package updates to one zone of the cloud at a time. On the first upgraded zone, we saw this issue on Octavia and elected to destroy the unit and redeploy as a workaround. Now, we've had the same issue recur on the second zone as well; I'm filing this bug as this does not appear to be an isolated issue.

"journalctl -xe -u ovs-vswitchd.service" shows the following output:

-----

May 10 20:18:52 juju-8de75b-6-lxd-9 systemd[1]: Starting Open vSwitch Forwarding Unit...
-- Subject: Unit ovs-vswitchd.service has begun start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit ovs-vswitchd.service has begun starting up.
May 10 20:18:52 juju-8de75b-6-lxd-9 ovs-ctl[54483]: nice: cannot set niceness: Permission denied
May 10 20:18:52 juju-8de75b-6-lxd-9 ovs-ctl[54483]: ovs-vswitchd: pthread_create failed (Resource temporarily unavailable)
May 10 20:18:52 juju-8de75b-6-lxd-9 ovs-vswitchd[54518]: ovs|00002|daemon_unix|ERR|fork child died before signaling startup (killed (Aborted), core dumped)
May 10 20:18:52 juju-8de75b-6-lxd-9 ovs-vswitchd[54518]: ovs|00003|daemon_unix|EMER|could not detach from foreground session
May 10 20:18:52 juju-8de75b-6-lxd-9 ovs-ctl[54483]: ovs-vswitchd: could not detach from foreground session
May 10 20:18:52 juju-8de75b-6-lxd-9 ovs-ctl[54483]: * Starting ovs-vswitchd
May 10 20:18:52 juju-8de75b-6-lxd-9 systemd[1]: ovs-vswitchd.service: Control process exited, code=exited status=1
May 10 20:18:52 juju-8de75b-6-lxd-9 systemd[1]: ovs-vswitchd.service: Failed with result 'exit-code'.
May 10 20:18:52 juju-8de75b-6-lxd-9 systemd[1]: Failed to start Open vSwitch Forwarding Unit.
-- Subject: Unit ovs-vswitchd.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit ovs-vswitchd.service has failed.
--
-- The result is RESULT.

-----

Because of the "pthread_create failed" message noted above, I am suspecting a connection with https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1830746, but I am not certain.

This cloud is a Bionic/Stein environment, and the Octavia unit is running version 2.11.5-0ubuntu0.19.04.1~cloud0 of openvswitch-{common,switch} and python3-openvswitch.

Thanks for any assistance you can provide.

Best Regards,
Paul Goins

[Billy Olsen (billy-olsen)]

This looks like https://bugs.launchpad.net/charm-ovn-chassis/+bug/1906280. I'll follow up on the packaging bits, as we were also exploring a package level change to avoid this having to be a charm managed solution at large. We were in discussion with the upstream with the right way to fix this in the packaging bits.

In the meantime, you should be able to add --no-mlockall to the ovs-vsctl opts in /etc/default/openvswitch-switch file to avoid this issue. You can also upgrade to the latest charms to get the charm managed fix.

[Billy Olsen (billy-olsen)]

I will confirm the package change bits that were in progress. Note, the packages that were fixed in bug 1906280 is around ensuring that the systemd unit files properly load information from the /etc/default/openvswitch-switch file (as they were not doing so previously). It does NOT address automatically specifying --no-mlockall when running inside a container.