[SRU] ovsdb-server.service needs a depedency on local-fs.target

Bug #1887177 reported by Brian Turek on 2020-07-10
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
openvswitch (Ubuntu)
Status tracked in Groovy
Focal
Undecided
Unassigned
Groovy
Undecided
Chris MacNaughton

Bug Description

[Impact]
When /var is on a separate filesystem (ZFS), ovsdb-server crashes on start if it is triggered before that filesystem is ready.

I recently just did a from-scratch install of Ubuntu 20.04 server edition and ran into issues with Open vSwitch and ZFS. I attempted to use ZFS for all of /var only to find that ovsdb-server pre-empted my ZFS /var mount which caused it to crash when trying to read its configuration DB at/var/lib/openvswitch/conf.db After much troubleshooting, the problem basically boils down to ovsdb-server.service needing a requirement on local-fs.target

I then found a blog post on Open Cloud Blog (https://www.opencloudblog.com/?p=240) that contained a fix:

The "After" line /lib/systemd/system/ovsdb-server.service needs the following changes:

    [Unit]
    Description=Open vSwitch Database Unit
    After=syslog.target network-pre.target dpdk.service local-fs.target
    Before=network.target networking.service
    PartOf=openvswitch-switch.service
    DefaultDependencies=no

    [Service]
    LimitNOFILE=1048576
    Type=forking
    Restart=on-failure
    EnvironmentFile=-/etc/default/openvswitch
    ExecStart=/usr/share/openvswitch/scripts/ovs-ctl \
              --no-ovs-vswitchd --no-monitor --system-id=random \
              start $OPTIONS
    ExecStop=/usr/share/openvswitch/scripts/ovs-ctl --no-ovs-vswitchd stop
    ExecReload=/usr/share/openvswitch/scripts/ovs-ctl --no-ovs-vswitchd \
               --no-monitor restart $OPTIONS
    RuntimeDirectory=openvswitch
    RuntimeDirectoryMode=0755

[Test Case]

Install ZFS on a machine, configure /var to be mounted on ZFS, install Open vSwitch, restart the server. The OpenvSwitch process should wait on the ZFS mount to start.

[Regression Potential]

Low. The only change in this is to defer the ovsdb-server startup until after the local-fs Systemd target has started. The only risk I can forsee is if the local-fs target didn't come up.

[racb] Service dependency and thus ordering is being adjusted, so if there is a regression it might manifest in users with unusual or different service installations from the norm, or in users with customised service configurations. There might also be unrelated latent issues or race conditions revealed as a result of changing the order of service startups.

[Discussion]

Related branches

Changed in openvswitch (Ubuntu):
status: New → In Progress
assignee: nobody → Chris MacNaughton (chris.macnaughton)

We are affected by this bug, too.

A more stable workaround than modifying /lib/systemd/system/ovsdb-server.service might be adding an overrides file (so that the fix will survive package upgrades).

On our affected systems, we added a file /etc/systemd/system/ovsdb-server.service.d/after-local-fs.conf with the following content:

[Unit]
After=local-fs.target

Corey Bryant (corey.bryant) wrote :

This is fixed in the groovy openvswitch packaging branch.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvswitch - 2.13.1-0ubuntu1

---------------
openvswitch (2.13.1-0ubuntu1) groovy; urgency=medium

  [ Chris MacNaughton ]
  * d/openvswitch-switch.ovsdb-server.service: Add local-fs.target to
    systemd service file to ensure that local filesystems are ready
    before the ovsdb service tries to start (LP: #1887177).
  * d/control: Remove Breaks/Replaces that are older than Focal (LP: #1878419).

  [ James Page ]
  * New upstream point release.
  * d/p/py3-compat.patch: Refresh.

 -- James Page <email address hidden> Wed, 05 Aug 2020 12:17:06 +0100

Changed in openvswitch (Ubuntu):
status: In Progress → Fix Released
Brian Turek (brian-turek) wrote :

Is this something that can be released to Focal as a SRU?

summary: - ovsdb-server.service needs a depedency on local-fs.target
+ [SRU] ovsdb-server.service needs a depedency on local-fs.target
description: updated
Robie Basak (racb) on 2020-09-30
description: updated

Hello Brian, or anyone else affected,

Accepted openvswitch into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvswitch/2.13.1-0ubuntu0.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openvswitch (Ubuntu Focal):
status: New → Fix Committed
tags: added: verification-needed verification-needed-focal
Brian Turek (brian-turek) wrote :

I'm not entirely sure what happened but it appears the build completely broke on amd64 with a majority of the tests failing. I fear the unrelated minor version bump (2.13.0 to 2.13.1) may have really gotten things in a bad state.

Brian Turek (brian-turek) wrote :

I checked proposed and the amd64 versions finally built. I installed v2.13.1-0ubuntu0.20.04.1, removed my temporary fix, rebooted, and am pleased to report that ovsdb-server is now waiting for ZFS to mount. Manual inspection using "systemctl list-dependencies ovsdb-server.service --after" shows dependencies on the relevant ZFS mounts as well.

tags: added: verification-done-focal
removed: verification-needed-focal
James Page (james-page) on 2020-10-12
tags: added: verification-done
removed: verification-needed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers