OVS 2.9+ systemd integration issues

Bug #1813371 reported by Alexandru Avadanii on 2019-01-26
This bug affects 5 people
Affects Status Importance Assigned to Milestone
openvswitch (Ubuntu)
James Page
James Page

Bug Description

For a few months now, we have been using OVS 2.9 (or newer) on Ubuntu Xenial in OPNFV, both with and without DPDK.
A while ago, we observed a couple of rare race conditions when multiple Linux interfaces/bridges are mixed with OVS ports/bridges. We also observed races between DPDK binding and openvswitch-switch (actually openvswitch-switch-dpdk configured using alternatives).
We worked around those issues by using a solution derived from the official OVS Debian readme, which recommends avoiding using `auto` for OVS bridges. Instead, we used `auto` for OVS bridges, but omitted the `auto` for the OVS ports in them. That worked almost perfectly for a while.

However, we recently bumped a few unrelated software components (since we migrated from Queens to Rocky in OPNFV) and we started experiecing race conditions again.

So I dugg a bit and found a couple of things:

1. Broken dependency between ovsdb-server/ovs-vswitchd systemd services and networking.service
This is probably a copy-pasta error from [1] `Before: network.service` which should probably be `Before: networking.service` on Debian systems.
The consequence is quite serious - on Debian systems, the OVS services start *after* networking.service.
Changing this leads to a service order change, which turns out to be quite the rabbit hole ...

2. Outdated ifupdown scripts
For example /etc/network/if-pre-up.d/openvswitch still references the old `openvswitch-nonetwork.service`.
Luckily, this is not critical, as the fallback uses `service openvswitch-switch [...]`, so I'm not sure this should be changed, but I thought it's worth mentioning.

3. Debian OVS does *not* handle OVS bridges without `auto`
Upstream OVS readme recommends ommitting `auto` for OVS bridges, as mentioned earlier, to avoid exactly the race conditions we saw.
Although following the recommendation in the upstream readme leads to a working system (`networking.service` no longer fails to start due to missing OVS bridges and/or vice-versa - ovs services no longer complain about Linux interfaces being in down state when trying to add them to OVS bridges), OVS bridges end up in DOWN state since nobody bothers to ifup them.
Imo, networking.service (or some *other* mechanism) should call `/sbin/ifup --allow=ovs -a --read-environment` *after* the initial `/sbin/ifup -a --read-enviroment` (provided the ordering issue #1 was changed to start OVS first, of course).

4. ovsdb-server should never start before DPDK service if DPDK is installed
This should actually be easy to fix and I have to admit I haven't run into it lately, although I remember it being an issue a while ago.
Anyway, a simple `After: dpdk.service` wouldn't hurt.

5. If OVS starts before networking.service, cloud-init causes cyclic dependencies
If we configure OVS services to start first, systemd might decide to randomly remove some units to break the following circular dependency:
  ovs-vswitchd --> ovsdb-server -(default dep)-> sysinit.target -->
  cloud-init.service --> networking.service --> ovs-vswitchd
In my tests, I just set 'DefaultDependencies=no' for OVS services, although this might require explicitly adding back some of the indirect dependencies of `sysinit.target`, so it's a sensible recommendation.

On my test systems, I didn't bother handling #2, as for the others I have some systemd drop-ins (see below), which so far seem to produce reproductible working environments.

# cat /etc/systemd/system/ovsdb-server.service.d/override.conf

# cat /etc/systemd/system/networking.service.d/ovs_workaround.conf
ExecStart=/sbin/ifup --allow=ovs -a --read-environment

# cat /etc/systemd/system/ovs-vswitchd.service.d/override.conf

# lsb_release -rd
Description: Ubuntu 16.04.5 LTS
Release: 16.04

# apt-cache policy openvswitch-switch
  Installed: 2.9.0-0ubuntu1~cloud0
  Candidate: 2.9.0-0ubuntu1~cloud0
  Version table:
 *** 2.9.0-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/queens/main amd64 Packages
        100 /var/lib/dpkg/status

[1] https://github.com/openvswitch/ovs/blob/master/rhel/usr_lib_systemd_system_ovsdb-server.service#L4

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openvswitch (Ubuntu):
status: New → Confirmed
James Page (james-page) on 2019-07-15
Changed in openvswitch (Ubuntu Eoan):
status: Confirmed → In Progress
assignee: nobody → James Page (james-page)
importance: Undecided → Medium
Changed in openvswitch (Ubuntu Disco):
importance: Undecided → Medium
Changed in openvswitch (Ubuntu Bionic):
importance: Undecided → Medium
Changed in openvswitch (Ubuntu Disco):
status: New → Triaged
Changed in openvswitch (Ubuntu Bionic):
status: New → Triaged
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvswitch - 2.12.0~git20190903.1cdf291dc-0ubuntu1

openvswitch (2.12.0~git20190903.1cdf291dc-0ubuntu1) eoan; urgency=medium

  * New upstream snapshot from 2.12 branch.
  * d/p/0001-acinclude-Also-use-LIBS-from-dpkg-pkg-config.patch: Drop,
    included upstream.
  * d/control: Bumped Standards-Version to 4.4.0.
  * d/control,rules: Drop Python 2 support.
  * d/rules: Disable testing of DPDK build on arm64 as builders don't
    have the required crc32 CPU feature.
  * d/control: Version BD for libdpdk-dev to ensure buid without overlinking.

 -- James Page <email address hidden> Fri, 06 Sep 2019 09:46:28 +0100

Changed in openvswitch (Ubuntu Eoan):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers