ebtables: Lock file handling has races

[Amad Ali]

[Impact]

 * ebtables uses creation of a file with an exclusive flag
   as a lock to synchronize table modification when used
   with --concurrent parameter.

 * If ebtables crashes it will leave a stale lock file.
   This will prevent another copy of ebtables from running,
   and indirectly any other service that depends on ebtables
   will also be affected.

 * This change adds support for real locks that get
   cleaned up if a process exits or crashes.

[Test Case]

 * Test Case1:
   1. $ sudo touch /var/lib/ebtables/lock"
   2. $ sudo ebtables --concurrent -L
   3. ebtables can't acquire a lock

 * Test Case 2:
   1. $ while true; do /usr/sbin/ebtables --concurrent -L; done
   2. hard reboot VM
   3. likely that the lock file is present under /var/lib/ebtables
   4. libvird hanging, try to connect to qemu:///system

[Regression Potential]

 * Normal Use:
   There is no regression potential during normal use and
   operation of ebtables.

 * Package Upgrade:
   There is a very very small regression potential during the package
   upgrade to the latest version. Once the package is upgraded that
   potential is gone. It is a very small potential because several
   things have to happen in a very small time frame and in an exact
   order since ebtables is not a resident program like a daemon:
     1. ebtables is launched during package upgrade AND
     2. new ebtables binary has not yet been written to disk AND
     3. it is launched with --concurent switch AND
     4. another ebtables with new binary is launched AND
     5. it is launched with --concurent switch AND
     6. the first ebtables copy hasn't exited yet AND
     7. both copies of ebtables are launched with a WRITE command AND
     8. both copies of ebtables are manipulating the same resource.
   Then one of the binaries could potentially fail, but once the old
   binary exits the potential is gone so subsequent re-runs of
   ebtables will succeed.

* Dragan's patch has been submitted to Debian via :
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860590

* Note that the ebtables upstream project is nearly dead. Nowadays, all the development is now happening in nft project which is intended to be replacement.

[Original Text]
libvirtd is hanging after startup due to ebtables lock file -from an earlier run- remains intact when the system reboots.
Same issue is happening than it is reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1290327 when the system boots.

After booting the system, It's not possible connect to the qemu-service.
- libvirt daemon tried to obtain a lock:
[pid 20966] read(24, "Trying to obtain lock /var/lib/e"..., 1024) = 45
[pid 20966] poll([{fd=22, events=POLLIN}, {fd=24, events=POLLIN}], 2, 4294967295) = 1 ([{fd=24, revents=POLLIN}])
[pid 20966] read(24, "Trying to obtain lock /var/lib/e"..., 1024) = 45
[pid 20966] poll([{fd=22, events=POLLIN}, {fd=24, events=POLLIN}], 2, 4294967295) = 1 ([{fd=24, revents=POLLIN}])
[pid 20966] read(24, "Trying to obtain lock /var/lib/e"..., 1024) = 45
[pid 20966] poll([{fd=22, events=POLLIN}, {fd=24, events=POLLIN}], 2, 4294967295^CProcess 20916 detached

- there was a file named 'lock' in /var/lib/ebtables directory with timestamp 14:54
- ebtables was configured:
* Ebtables support available, number of installed rules [ OK ]
(other nodes appeared to be in the same state from ebtables point of view, but without the lock file)
- I removed the lock file and libvirt started to work instantly - the lock obtain messages have disappeared from the trace and virsh commands are working
- at 14:54 the host was booting up. According to the logs, there were other reboots after that one, but the lock file remained intact (at least the timestamp was not updated).

Could you please suggest a solution to be sure that ebtables lock file is removed during boot?

[Amad Ali (amad)]

Created a hotfix ebtables (2.0.10.4-3ubuntu2~lp1645324hotfix3) in the following PPA.

https://launchpad.net/~amad/+archive/ubuntu/ericsson/+packages

The hotfix is verified by zsolt-babindai (<email address hidden>).

Attaching the patch here to request the start the SRU process.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "debdiff-ebtables2.0.10.4-3ubuntu2" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Christian Ehrhardt  (paelzer)]

Hi,
thank you for providing a patch for SRU review.

I wanted to let you know that you should also add a SRU Template to not be blocked on that when the SRU Team does the review.

Please see https://wiki.ubuntu.com/StableReleaseUpdates for the details of the process.
You usually copy the template that is on there and modify the placeholders to what applies to your case.

[Nish Aravamudan (nacc)]

Fixed in Debian in 2.0.10.4-3.3, which is present in Xenial (2.0.10.4-3.4ubuntu1) and on.

[Nish Aravamudan (nacc)]

Note that also, on quick review of the debdiff, which is a backport of the Debian change, I would suggest a version of '2.0.10.4-3ubuntu1.1' rather than '2.0.10.4-3ubuntu2'.

[Amad Ali (amad)]

Attaching the debdiff with the changes requested.

[Robie Basak (racb)]

What happens if:

1) ebtables is running at the time of an SRU update? Then a new ebtables would fail to detect the lock of the old ebtables invocation.

2) Existing scripts depend on the location of the ebtables lock?

For the SRU, would it make more sense to not move the lock path, and instead arrange for /var/lib/ebtables to be cleared on boot? Opinions welcome.

[Robie Basak (racb)]

Unsubscribing ~ubuntu-sponsors as this is in the queue.

[Robie Basak (racb)]

Please add your conclusions to Regression Potential if you'd still like to proceed with this.

[Dragan S. (dragan-s)]

Good points racb, I think the startup script should test on "start" the existence of the process and the lock. If process is not running it should remove the lock before starting a new process. Also file should be removed on "stop".

[Robie Basak (racb)]

I suggest patching Trusty to remove the lock file on boot, before ebtables has any chance to run, if that's possible. That seems to me to be the minimal fix that would resolve any user impact on Trusty. Using /run is a better way to do that, and in the future we'll do it that way (since Zesty has this fixed) but for a stable release update we prefer the minimal and least disruptive fix over a "correct" one.

[Dragan S. (dragan-s)]

Creating a "fix" that only removes the file on boot is not a fix....

I have taken over this issue from Amad and have looked into the code. The file exclusive create approach is a bad substitution for real locking. I have a patch done that I am in the process of building a PPA to test, this uses real locking.

File existence as a lock is bad because if the process dies or is killed it leaves a stale file around blocking any new ebtables instance from running. The approach taken in Zesty is not a real solution either since it just shifts the problem around. If the process is killed or crashes by an un-trappable signal the file will persist until the system is rebooted. Hardly a solution....

My patch creates a real lock on the file that is automatically released on exit or a crash.

[Robie Basak (racb)]

> Creating a "fix" that only removes the file on boot is not a fix....

Sure, but if it fixes (and is expected to fix completely) the actual impact to users then it may be the most appropriate for an SRU. We don't fix theoretical bugs in SRUs.

[Dragan S. (dragan-s)]

> Sure, but if it fixes (and is expected to fix completely) the actual impact
> to users then it may > be the most appropriate for an SRU. We don't fix
> theoretical bugs in SRUs.

As far as I understand the impact to the user was that a lock file from a previous run prevents ebtables to run again. And then second part was that user tried to reboot and that didn't help either. So a reboot is not the fix part, the fix part is in ebtables to not leave stale locks around...

[Dragan S. (dragan-s)]

Use real locking in ebtables (LP: #1645324)
    - Prior use of locking by file exclusive access is inadequate
      because if ebtables crashes or is killed it will leave a
      stale lock file behind which then blocks new ebtables from
      running.

[Louis Bouchard (louis)]

Hi dragan-s,

This fix should be sent upstream for inclusion in the official project.

...Louis

[Robie Basak (racb)]

@Dragan,

Thank you for your reasoning in comment 18. This makes sense.

Have you sent the real locking fix upstream please, and has there been any response? I see this in the Trusty queue, but doesn't look like this is fixed in Z+1 yet. I guess that wants updating and uploading as soon as Z+1 opens?

[Robie Basak (racb)]

A dep3 "Forwarded" header would be most useful!

[Robie Basak (racb)]

As the conclusion in this bug is that better lock handling is needed (not just a path move), I've renamed the bug, and the fix isn't in the development release yet so I changed that back to Triaged.

[Dragan S. (dragan-s)]

@Robie-

Hi Robie-

if by Z+1 you mean Zesty, I have attached the patch for zesty and Dan nominated it for zesty but we are waiting for someone with permissions to confirm it.

I also filed a bug with upstream Debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860590

The DEP-3 header is in the diff:

+Description: Use real locking in ebtables
+ Prior use of locking by file exclusive access is inadequate
+ because if ebtables crashes or is killed it will leave a
+ stale lock file behind which then blocks new ebtables from
+ running.
+Author: <email address hidden>
+Bug: https://bugs.launchpad.net/ubuntu/+source/ebtables/+bug/1645324
+---
+This patch header follows DEP-3: http://dep.debian.net/deps/dep3/
+Index: ebtables-2.0.10.4/libebtc.c

Is that what you are looking for?

Thanks

[Dragan S. (dragan-s)]

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860590

[Christian Ehrhardt  (paelzer)]

Hi,
Zesty+1 meant the following release which is in the meantime known as "Artful".
For the SRU handling [1] in general changes need to go to the current development release first and then SRU'ed back. When you worked on this the last time zesty was just about to be released, but artful not yet open - now it is.

Thanks for linking the Debian bug as well!

Also was there any update/response from upstream one can consider when checking the locking rewrite?

[Eric Desrochers (slashd)]

@ChristianEhrhardt & @Robie Basak

I'll be covering Dragan for this bug from now on.

I'm afraid that there is not much development activities in the ebtables upstream project nowadays :

Last 3 ebtables commits :
Date: Tue Mar 28 19:42:39 2017 +0530
Date: Thu Oct 22 16:41:03 2015 +0200
Date: Sat May 16 12:31:58 2015 +0200

I'll try to reach the netfilter team, and see if they are still accepting patches for this project.
In case of the upstream project is "dead", do you think there is a chance Dragan's fix can land in Ubuntu anyway ?

- Eric

[Eric Desrochers (slashd)]

Additionally, it seems like it isn't the first time Debian patched ebtables, thus Ubuntu package without submitting upstream.

Example : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813760

Zesty :
libebtc.c:137:#define LOCKDIR "/run"
libebtc.c:138:#define LOCKFILE LOCKDIR"/ebtables.lock"

Zesty changelog :
...
  * Non-maintainer upload.
  ...
  * debian/patches/lockdirfix.patch: Move the lock file to /run instead of
    /var/lib/ebtables directory (Closes: #813760)
...
 -- Laurent Bigonville <email address hidden> Fri, 05 Feb 2016 13:29:39 +0100

Xenial :
x/ebtables-2.0.10.4/libebtc.c:137:#define LOCKDIR "/run"
x/ebtables-2.0.10.4/libebtc.c:138:#define LOCKFILE LOCKDIR"/ebtables.lock"

ebtables git repo :

Still point to "/var/lib/ebtables" for LOCKDIR, where debian changed it in 2016 to "/run".

libebtc.c:137:#define LOCKDIR "/var/lib/ebtables"
libebtc.c:138:#define LOCKFILE LOCKDIR"/lock"

From what I see Debian already abandonned to keep the ebtables project maintainer in the loop.

[Eric Desrochers (slashd)]

Artful debdiff

[Eric Desrochers (slashd)]

ebtables (for IPv4, IPv6, ARP and Ethernet bridging, respectively) are intended to be replaced with nft as a single unified implementation. With that being said, nowadays all the development/focus is happening in nft instead.

- Eric

[Eric Desrochers (slashd)]

<slashd> rbasak, morning I see you talked with dragan about this LP: #1645324. From now on, I'm taking care of this bug, I see that some ppls suggest Dragan to submit in the upstream project first, but in this case it seems that ebtables is nearly "dead" last commits were made in 2015. Nowadays, all the development happen in nft. Do you think Dragan's patch could be eligilble for SRu anyway ? considering that Dragan already submitted the same patch to Debian ebtables but not upstream.

<rbasak> slashd: can we get the patch committed/uploaded into Debian? That would also alleviate Ubuntu's maintenance burden.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ebtables - 2.0.10.4-3.5ubuntu2

---------------
ebtables (2.0.10.4-3.5ubuntu2) artful; urgency=medium

  * Use real locking in ebtables (LP: #1645324)
    - Prior use of locking by file exclusive access is inadequate
      because if ebtables crashes or is killed it will leave a
      stale lock file behind which then blocks new ebtables from
      running.

 -- Dragan Stancevic <email address hidden> Tue, 18 Apr 2017 17:32:38 -0500

[Łukasz Zemczak (sil2100)]

Hello Amad, or anyone else affected,

Accepted ebtables into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ebtables/2.0.10.4-3.5ubuntu1.17.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Amad, or anyone else affected,

Accepted ebtables into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ebtables/2.0.10.4-3.5ubuntu1.16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Amad, or anyone else affected,

Accepted ebtables into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ebtables/2.0.10.4-3.4ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Łukasz Zemczak (sil2100)]

Hello Amad, or anyone else affected,

Accepted ebtables into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ebtables/2.0.10.4-3ubuntu1.14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Szilard Cserey (szilard.cserey)]

Hi All,

I have successfully tested the following versions of ebtables

for Trusty: 2.0.10.4-3ubuntu1.14.04.1 OK

for Xenial: 2.0.10.4-3.4ubuntu2 OK

for Yakkety: 2.0.10.4-3.5ubuntu1.16.10.1 OK

for Zesty: 2.0.10.4-3.5ubuntu1.17.04.1 OK

Szilard

[Eric Desrochers (slashd)]

@szilard, is your testing/verification has been made based on Dragan's test case #1 and #2 ?

Could you please provide more details ? output ....

This will be useful informations for the SRU members to give their final approval.

- Eric

[Szilard Cserey (szilard.cserey)]

The verification was done using Test Case 1 and Test Case 2

However I noticed that in the Trusty version the lock file
is located at /var/lib/ebtables/lock
Whereas on the Xenial/Yakkety/Zesty versions lock file is
at /run/ebtables.lock

[Eric Desrochers (slashd)]

@szilard,

The constant values (#define LOCKDIR ...) hasn't been touched in these patches, meaning that it was already like that before this SRU.

With that being said, the LOCKDIR location isn't a problem, so IMHO, there is no need for the LOCKDIR in Trusty to move to "/run" if it doesn't create a problem.

The real issue here is that if ebtables crashes or is killed it leave a stale lock file behind which then blocks new ebtables from running.

This SRU is precisely fixing this issue by improving the locking mechanism.

- Eric

[Eric Desrochers (slashd)]

It has been brought to my attention the following :

"I was able to install and test it and seems to be OK."

^ Tested by someone affected by the issue using the trusty-proposed package.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ebtables - 2.0.10.4-3.5ubuntu1.17.04.1

---------------
ebtables (2.0.10.4-3.5ubuntu1.17.04.1) zesty; urgency=medium

  * Use real locking in ebtables (LP: #1645324)
    - Prior use of locking by file exclusive access is inadequate
      because if ebtables crashes or is killed it will leave a
      stale lock file behind which then blocks new ebtables from
      running.

 -- Dragan Stancevic <email address hidden> Tue, 18 Apr 2017 17:32:38 -0500

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for ebtables has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ebtables - 2.0.10.4-3.5ubuntu1.16.10.1

---------------
ebtables (2.0.10.4-3.5ubuntu1.16.10.1) yakkety; urgency=medium

  * Use real locking in ebtables (LP: #1645324)
    - Prior use of locking by file exclusive access is inadequate
      because if ebtables crashes or is killed it will leave a
      stale lock file behind which then blocks new ebtables from
      running.

 -- Dragan Stancevic <email address hidden> Tue, 18 Apr 2017 17:32:38 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package ebtables - 2.0.10.4-3.4ubuntu2

---------------
ebtables (2.0.10.4-3.4ubuntu2) xenial; urgency=medium

  * Use real locking in ebtables (LP: #1645324)
    - Prior use of locking by file exclusive access is inadequate
      because if ebtables crashes or is killed it will leave a
      stale lock file behind which then blocks new ebtables from
      running.

 -- Dragan Stancevic <email address hidden> Mon, 29 May 2017 16:07:53 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package ebtables - 2.0.10.4-3ubuntu1.14.04.1

---------------
ebtables (2.0.10.4-3ubuntu1.14.04.1) trusty; urgency=medium

  * Use real locking in ebtables (LP: #1645324)
    - Prior use of locking by file exclusive access is inadequate
      because if ebtables crashes or is killed it will leave a
      stale lock file behind which then blocks new ebtables from
      running.

 -- Dragan Stancevic <email address hidden> Thu, 02 Mar 2017 17:34:55 -0600