seccomp_rule_add is very slow

Bug #1861177 reported by Sam Whited
28
This bug affects 3 people
Affects Status Importance Assigned to Milestone
libseccomp (Ubuntu)
Medium
Ioanna Alifieraki
Xenial
Medium
Ioanna Alifieraki
Bionic
Medium
Ioanna Alifieraki
Eoan
Medium
Ioanna Alifieraki
Focal
Medium
Ioanna Alifieraki
Groovy
Medium
Ioanna Alifieraki

Bug Description

[IMPACT]
There is a known and patched issue with version 2.4 of libseccomp where certain operations have a large performance regression. This is causing some packages that use libseccomp such as container orchestration systems to occasionally time out or otherwise fail under certain workloads.

Please consider porting the patch into the various Ubuntu versions that have version 2.4 of libseccomp and into the backports. The performance patch from version 2.5 (yet to be released) applies cleanly on top of the 2.4 branch of libseccomp.

For more information, and for a copy of the patch (which can also be cherry picked from the upstream libseccomp repos) see the similar Debian issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943913

Upstream issue : https://github.com/seccomp/libseccomp/issues/153
Upstream fix : https://github.com/seccomp/libseccomp/pull/180/

[Test Case]

For this test case we use Docker on Ubuntu Groovy (20.10) :

--> Current libseccomp version
#dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3 amd64 high level interface to Linux seccomp filter

## pull ubuntu image
# docker pull ubuntu
## create a container
# docker run --name test_seccomp -it 74435f89ab78 /bin/bash

## run test case
# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
...
MAX TIME :
real 0m10,319s
user 0m0,018s
sys 0m0,033s

--> Patched libseccomp version

# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu4 amd64 high level interface to Linux seccomp filter

# docker start test_seccomp
## run test case
# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
...
MAX TIME :
real 0m3,650s
user 0m0,025s
sys 0m0,028s

[Regression Potential]

The first of the 2 patches cleans up the code that adds rules to a single filter without changing the logic of the code. The second patch introduces the idea of shadow transactions. On a successful transaction commit the old transaction checkpoint is preserved and is brought up to date with the current filter. The next time a new transaction starts, it checks is the a shadow transaction exist and if so the shadow is used instead of creating a new checkpoint from scratch [1]. This is the patch that mitigates the performance regression. Any potential regression will involve the parts of the code that add rules to filters and/or the code that creates and checks the shadow transactions.

[Other]

Affected releases : Groovy, Focal, Eoan, Bionic, Xenial.

[1] https://github.com/seccomp/libseccomp/pull/180/commits/bc3a6c0453b0350ee43e4925482f705a2fbf5a4d

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

@mvo and @ijohnson, fyi, the fix for this may help with slow snap-seccomp (unconfirmed; not actively working on it at this time).

Revision history for this message
Bryce Harrington (bryce) wrote :

The patch mentioned in the OP is attached for reference. Per [1] it was proposed for inclusion in Debian last November as patches db-consolidate-some-of-the-code-which-adds-rules-to-.patch and db-add-shadow-transactions.patch.

The corresponding bug report upstream is [2].

1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943913
2: https://github.com/seccomp/libseccomp/issues/153

Revision history for this message
Bryce Harrington (bryce) wrote :

2.4.1 is currently available from xenial, bionic, disco, eoan; focal carries 2.4.2. None of these carry the patches for this bug report yet.

Changed in libseccomp (Ubuntu):
importance: Undecided → High
status: New → Triaged
Bryce Harrington (bryce)
tags: added: server-next
tags: added: patch
Revision history for this message
John Lenton (chipaca) wrote :

I'm leaving the snapd bug task as New so mvo or ian can take a look when it's their triage day, at the latest; I have nothing useful to do here other than "ouch".

Michael Vogt (mvo)
Changed in snapd:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Ian Johnson (anonymouse67) wrote :

I'll take a look at measuring this with snapd $SOON

Changed in snapd:
assignee: nobody → Ian Johnson (anonymouse67)
Revision history for this message
Sam Whited (samwhited) wrote :

Gentle ping. Can this be assigned to someone? Thanks!

Revision history for this message
Ian Johnson (anonymouse67) wrote :

I can't seem to assign this bug to Dimitri, but as per https://github.com/snapcore/core20/issues/48, Dimitri should be preparing a libseccomp 2.4.2 SRU.

Changed in libseccomp (Ubuntu):
importance: High → Medium
status: Triaged → In Progress
Revision history for this message
Robie Basak (racb) wrote :

I asked Dimitri about this on IRC. He isn't expecting to work on this, and nor is the server team. We think the security team will take this on according to their priorities? If this does need the server team's attention, please let us know.

tags: removed: server-next
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

FYI, a 2.4.3 SRU is in flight (by amurray), but looking at https://github.com/seccomp/libseccomp/pull/180 (the fix for the bug), https://github.com/seccomp/libseccomp/issues/187 (2.4.3 backports), and code inspection, the fix for the bug is not in 2.4.3 and will come in 2.5.

The security team is not currently working on this, but that could be changed (that should be discussed outside of this bug).

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

There isn't a snapd task (snap-seccomp is compiled against libseccomp but it can't influence this behavior), so unassigning Ian and marking that task as Invalid.

Changed in snapd:
assignee: Ian Johnson (anonymouse67) → nobody
status: Triaged → Invalid
Revision history for this message
Sam Whited (samwhited) wrote :

Thanks for the follow ups. If the server team doesn't want to take this on, could someone provide me with a team or person I could contact to try and gently prod this forward (maybe someone on the security team)? I've got some customers that are running into this and telling them to rebuild this library themselves doesn't seem to have made them happy :)

Changed in libseccomp (Ubuntu Focal):
status: New → In Progress
Changed in libseccomp (Ubuntu Eoan):
status: New → In Progress
Changed in libseccomp (Ubuntu Bionic):
status: New → In Progress
Changed in libseccomp (Ubuntu Xenial):
status: New → In Progress
Changed in libseccomp (Ubuntu Focal):
importance: Undecided → Medium
Changed in libseccomp (Ubuntu Eoan):
importance: Undecided → Medium
Changed in libseccomp (Ubuntu Bionic):
importance: Undecided → Medium
Changed in libseccomp (Ubuntu Xenial):
importance: Undecided → Medium
assignee: nobody → Ioanna Alifieraki (joalif)
Changed in libseccomp (Ubuntu Bionic):
assignee: nobody → Ioanna Alifieraki (joalif)
Changed in libseccomp (Ubuntu Eoan):
assignee: nobody → Ioanna Alifieraki (joalif)
Changed in libseccomp (Ubuntu Focal):
assignee: nobody → Ioanna Alifieraki (joalif)
Changed in libseccomp (Ubuntu Groovy):
assignee: nobody → Ioanna Alifieraki (joalif)
description: updated
Revision history for this message
Ioanna Alifieraki (joalif) wrote :

Debdiff for Groovy.

Currently, there is an ongoing sru of libseccomp for F,E,B and X stuck in proposed.
Once it is released in updates, I'll sru this one for F,E,B and X.

Revision history for this message
Ioanna Alifieraki (joalif) wrote :

Debdiff for Focal.

Revision history for this message
Ioanna Alifieraki (joalif) wrote :

Debdiff for Eoan.

Revision history for this message
Ioanna Alifieraki (joalif) wrote :

Debdiff for Bionic.

Revision history for this message
Ioanna Alifieraki (joalif) wrote :

Debdiff for Xenial.

Revision history for this message
Dan Streetman (ddstreet) wrote :

uploaded to x/b/e/f/g, thanks!

Revision history for this message
Ioanna Alifieraki (joalif) wrote :

Currently, there are autopkgtest regressions on Groovy for the docker.io package :
https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#libseccomp

These regressions are not due to the new libseccomp (can be reproduced with previous libseccomp version). They are due to a bug in cloud-init when in lxd container, where 'cloud-init status' status returns error. See bug https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1886531

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

As discussed on IRC, I have reviewed all the SRUs. They looked correct from my POV (although I guess I'd appreciate a second opinion from the security team).

Since we want to include those in the -security pockets as well, I have uploaded all of them to a security-only Bileto PPA. I have confirmed via the build logs that only -security is enabled. The PPA can be found here:

https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4143/+packages

Once the packages build correctly, I will bin-copy them to -proposed and remove the source uploads from the upload queues.

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Sam, or anyone else affected,

Accepted libseccomp into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libseccomp/2.4.3-1ubuntu3.20.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in libseccomp (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Sam, or anyone else affected,

Accepted libseccomp into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libseccomp/2.4.3-1ubuntu3.19.10.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in libseccomp (Ubuntu Eoan):
status: In Progress → Fix Committed
tags: added: verification-needed-eoan
Changed in libseccomp (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Sam, or anyone else affected,

Accepted libseccomp into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libseccomp/2.4.3-1ubuntu3.18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in libseccomp (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed-xenial
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello Sam, or anyone else affected,

Accepted libseccomp into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libseccomp/2.4.3-1ubuntu3.16.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (libseccomp/2.4.3-1ubuntu3.19.10.3)

All autopkgtests for the newly accepted libseccomp (2.4.3-1ubuntu3.19.10.3) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/242-7ubuntu3.11 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#libseccomp

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (libseccomp/2.4.3-1ubuntu3.18.04.3)

All autopkgtests for the newly accepted libseccomp (2.4.3-1ubuntu3.18.04.3) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

lxc/3.0.3-0ubuntu1~18.04.1 (arm64)
systemd/unknown (armhf)
systemd/237-3ubuntu10.41 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#libseccomp

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libseccomp - 2.4.3-1ubuntu4

---------------
libseccomp (2.4.3-1ubuntu4) groovy; urgency=medium

  * d/p/db-consolidate-some-of-the-code-which-adds-rules.patch
  * d/p/db-add-shadow-transactions.patch (LP: #1861177)
    Backport upstream patches to address performance regression introduced
    in libseccomp 2.4.

 -- Ioanna Alifieraki <email address hidden> Mon, 22 Jun 2020 11:10:27 +0100

Changed in libseccomp (Ubuntu Groovy):
status: In Progress → Fix Released
Revision history for this message
Ioanna Alifieraki (joalif) wrote :

#VERIFICATION XENIAL

---> Old version
# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.16.04.2 amd64 high level interface to Linux seccomp filte

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 8.923s

---> New version

# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.16.04.3 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 3.270s

tags: added: verification-done-xenial
removed: verification-needed-xenial
Revision history for this message
Ioanna Alifieraki (joalif) wrote :

#VERIFICATION FOCAL

---> Old version
# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.20.04.2 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 10.176s

---> New version

# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.20.04.3 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 3.161s

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Ioanna Alifieraki (joalif) wrote :

#VERIFICATION BIONIC

---> Old version
# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.18.04.2 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 8.148s

---> New version

# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.18.04.3 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 2.713s

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Ioanna Alifieraki (joalif) wrote :

#VERIFICATION EOAN

---> Old version
# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.19.10.2 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 9.374s

---> New version

# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.19.10.3 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 2.009s

tags: added: verification-done-eoan
removed: verification-needed-eoan
Revision history for this message
Ioanna Alifieraki (joalif) wrote :

# autopkgtest regression in EOAN

There is an autopkgtest regression in EOAN on ppc64el for systemd package.
The regression is caused by the 'upstream' test.
As it's shown here http://autopkgtest.ubuntu.com/packages/s/systemd/eoan/ppc64el this test
has always been failing since systemd version 240-6ubuntu5 and can be ignored.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libseccomp - 2.4.3-1ubuntu3.20.04.3

---------------
libseccomp (2.4.3-1ubuntu3.20.04.3) focal; urgency=medium

  * d/p/db-consolidate-some-of-the-code-which-adds-rules.patch
  * d/p/db-add-shadow-transactions.patch (LP: #1861177)
    Backport upstream patches to address performance regression introduced
    in libseccomp 2.4.

 -- Ioanna Alifieraki <email address hidden> Mon, 29 Jun 2020 13:35:12 +0100

Changed in libseccomp (Ubuntu Focal):
status: Fix Committed → Fix Released
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for libseccomp has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libseccomp - 2.4.3-1ubuntu3.18.04.3

---------------
libseccomp (2.4.3-1ubuntu3.18.04.3) bionic; urgency=medium

  * d/p/db-consolidate-some-of-the-code-which-adds-rules.patch
  * d/p/db-add-shadow-transactions.patch (LP: #1861177)
    Backport upstream patches to address performance regression introduced
    in libseccomp 2.4.

 -- Ioanna Alifieraki <email address hidden> Mon, 29 Jun 2020 13:52:22 +0100

Changed in libseccomp (Ubuntu Bionic):
status: Fix Committed → Fix Released
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libseccomp - 2.4.3-1ubuntu3.16.04.3

---------------
libseccomp (2.4.3-1ubuntu3.16.04.3) xenial; urgency=medium

  * d/p/db-consolidate-some-of-the-code-which-adds-rules.patch
  * d/p/db-add-shadow-transactions.patch (LP: #1861177)
    Backport upstream patches to address performance regression introduced
    in libseccomp 2.4.

 -- Ioanna Alifieraki <email address hidden> Mon, 29 Jun 2020 13:57:55 +0100

Changed in libseccomp (Ubuntu Xenial):
status: Fix Committed → Fix Released
status: Fix Committed → Fix Released
Revision history for this message
Dariusz Gadomski (dgadomski) wrote :

Marking Eoan as Won't fix due to EOL.

Changed in libseccomp (Ubuntu Eoan):
status: Fix Committed → Won't Fix
Mathew Hodson (mhodson)
affects: snapd → ubuntu-translations
no longer affects: ubuntu-translations
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libseccomp - 2.4.3-1ubuntu3.19.10.3

---------------
libseccomp (2.4.3-1ubuntu3.19.10.3) eoan; urgency=medium

  * d/p/db-consolidate-some-of-the-code-which-adds-rules.patch
  * d/p/db-add-shadow-transactions.patch (LP: #1861177)
    Backport upstream patches to address performance regression introduced
    in libseccomp 2.4.

 -- Ioanna Alifieraki <email address hidden> Mon, 29 Jun 2020 13:47:58 +0100

Changed in libseccomp (Ubuntu Eoan):
status: Won't Fix → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.