seccomp_rule_add is very slow

[Sam Whited]

[IMPACT]
There is a known and patched issue with version 2.4 of libseccomp where certain operations have a large performance regression. This is causing some packages that use libseccomp such as container orchestration systems to occasionally time out or otherwise fail under certain workloads.

Please consider porting the patch into the various Ubuntu versions that have version 2.4 of libseccomp and into the backports. The performance patch from version 2.5 (yet to be released) applies cleanly on top of the 2.4 branch of libseccomp.

For more information, and for a copy of the patch (which can also be cherry picked from the upstream libseccomp repos) see the similar Debian issue: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943913

Upstream issue : https://github.com/seccomp/libseccomp/issues/153
Upstream fix : https://github.com/seccomp/libseccomp/pull/180/

[Test Case]

For this test case we use Docker on Ubuntu Groovy (20.10) :

--> Current libseccomp version
#dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3 amd64 high level interface to Linux seccomp filter

## pull ubuntu image
# docker pull ubuntu
## create a container
# docker run --name test_seccomp -it 74435f89ab78 /bin/bash

## run test case
# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
...
MAX TIME :
real 0m10,319s
user 0m0,018s
sys 0m0,033s

--> Patched libseccomp version

# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu4 amd64 high level interface to Linux seccomp filter

# docker start test_seccomp
## run test case
# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
...
MAX TIME :
real 0m3,650s
user 0m0,025s
sys 0m0,028s

[Regression Potential]

The first of the 2 patches cleans up the code that adds rules to a single filter without changing the logic of the code. The second patch introduces the idea of shadow transactions. On a successful transaction commit the old transaction checkpoint is preserved and is brought up to date with the current filter. The next time a new transaction starts, it checks is the a shadow transaction exist and if so the shadow is used instead of creating a new checkpoint from scratch [1]. This is the patch that mitigates the performance regression. Any potential regression will involve the parts of the code that add rules to filters and/or the code that creates and checks the shadow transactions.

[Other]

Affected releases : Groovy, Focal, Eoan, Bionic, Xenial.

[1] https://github.com/seccomp/libseccomp/pull/180/commits/bc3a6c0453b0350ee43e4925482f705a2fbf5a4d

[Jamie Strandboge (jdstrand)]

@mvo and @ijohnson, fyi, the fix for this may help with slow snap-seccomp (unconfirmed; not actively working on it at this time).

[Bryce Harrington (bryce)]

The patch mentioned in the OP is attached for reference. Per [1] it was proposed for inclusion in Debian last November as patches db-consolidate-some-of-the-code-which-adds-rules-to-.patch and db-add-shadow-transactions.patch.

The corresponding bug report upstream is [2].

1: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943913
2: https://github.com/seccomp/libseccomp/issues/153

[Bryce Harrington (bryce)]

2.4.1 is currently available from xenial, bionic, disco, eoan; focal carries 2.4.2. None of these carry the patches for this bug report yet.

[John Lenton (chipaca)]

I'm leaving the snapd bug task as New so mvo or ian can take a look when it's their triage day, at the latest; I have nothing useful to do here other than "ouch".

[Ian Johnson (anonymouse67)]

I'll take a look at measuring this with snapd $SOON

[Sam Whited (samwhited)]

Gentle ping. Can this be assigned to someone? Thanks!

[Ian Johnson (anonymouse67)]

I can't seem to assign this bug to Dimitri, but as per https://github.com/snapcore/core20/issues/48, Dimitri should be preparing a libseccomp 2.4.2 SRU.

[Robie Basak (racb)]

I asked Dimitri about this on IRC. He isn't expecting to work on this, and nor is the server team. We think the security team will take this on according to their priorities? If this does need the server team's attention, please let us know.

[Jamie Strandboge (jdstrand)]

FYI, a 2.4.3 SRU is in flight (by amurray), but looking at https://github.com/seccomp/libseccomp/pull/180 (the fix for the bug), https://github.com/seccomp/libseccomp/issues/187 (2.4.3 backports), and code inspection, the fix for the bug is not in 2.4.3 and will come in 2.5.

The security team is not currently working on this, but that could be changed (that should be discussed outside of this bug).

[Jamie Strandboge (jdstrand)]

There isn't a snapd task (snap-seccomp is compiled against libseccomp but it can't influence this behavior), so unassigning Ian and marking that task as Invalid.

[Sam Whited (samwhited)]

Thanks for the follow ups. If the server team doesn't want to take this on, could someone provide me with a team or person I could contact to try and gently prod this forward (maybe someone on the security team)? I've got some customers that are running into this and telling them to rebuild this library themselves doesn't seem to have made them happy :)

[Ioanna Alifieraki (joalif)]

Debdiff for Groovy.

Currently, there is an ongoing sru of libseccomp for F,E,B and X stuck in proposed.
Once it is released in updates, I'll sru this one for F,E,B and X.

[Ioanna Alifieraki (joalif)]

Debdiff for Focal.

[Ioanna Alifieraki (joalif)]

Debdiff for Eoan.

[Ioanna Alifieraki (joalif)]

Debdiff for Bionic.

[Ioanna Alifieraki (joalif)]

Debdiff for Xenial.

[Dan Streetman (ddstreet)]

uploaded to x/b/e/f/g, thanks!

[Ioanna Alifieraki (joalif)]

Currently, there are autopkgtest regressions on Groovy for the docker.io package :
https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#libseccomp

These regressions are not due to the new libseccomp (can be reproduced with previous libseccomp version). They are due to a bug in cloud-init when in lxd container, where 'cloud-init status' status returns error. See bug https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1886531

[Łukasz Zemczak (sil2100)]

As discussed on IRC, I have reviewed all the SRUs. They looked correct from my POV (although I guess I'd appreciate a second opinion from the security team).

Since we want to include those in the -security pockets as well, I have uploaded all of them to a security-only Bileto PPA. I have confirmed via the build logs that only -security is enabled. The PPA can be found here:

https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4143/+packages

Once the packages build correctly, I will bin-copy them to -proposed and remove the source uploads from the upload queues.

[Łukasz Zemczak (sil2100)]

Hello Sam, or anyone else affected,

Accepted libseccomp into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libseccomp/2.4.3-1ubuntu3.20.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Sam, or anyone else affected,

Accepted libseccomp into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libseccomp/2.4.3-1ubuntu3.19.10.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Sam, or anyone else affected,

Accepted libseccomp into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libseccomp/2.4.3-1ubuntu3.18.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Sam, or anyone else affected,

Accepted libseccomp into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libseccomp/2.4.3-1ubuntu3.16.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted libseccomp (2.4.3-1ubuntu3.19.10.3) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/242-7ubuntu3.11 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#libseccomp

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted libseccomp (2.4.3-1ubuntu3.18.04.3) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

lxc/3.0.3-0ubuntu1~18.04.1 (arm64)
systemd/unknown (armhf)
systemd/237-3ubuntu10.41 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#libseccomp

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package libseccomp - 2.4.3-1ubuntu4

---------------
libseccomp (2.4.3-1ubuntu4) groovy; urgency=medium

  * d/p/db-consolidate-some-of-the-code-which-adds-rules.patch
  * d/p/db-add-shadow-transactions.patch (LP: #1861177)
    Backport upstream patches to address performance regression introduced
    in libseccomp 2.4.

 -- Ioanna Alifieraki <email address hidden> Mon, 22 Jun 2020 11:10:27 +0100

[Ioanna Alifieraki (joalif)]

#VERIFICATION XENIAL

---> Old version
# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.16.04.2 amd64 high level interface to Linux seccomp filte

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 8.923s

---> New version

# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.16.04.3 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 3.270s

[Ioanna Alifieraki (joalif)]

#VERIFICATION FOCAL

---> Old version
# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.20.04.2 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 10.176s

---> New version

# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.20.04.3 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 3.161s

[Ioanna Alifieraki (joalif)]

#VERIFICATION BIONIC

---> Old version
# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.18.04.2 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 8.148s

---> New version

# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.18.04.3 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 2.713s

[Ioanna Alifieraki (joalif)]

#VERIFICATION EOAN

---> Old version
# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.19.10.2 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 9.374s

---> New version

# dpkg -l | grep libseccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.19.10.3 amd64 high level interface to Linux seccomp filter

# for i in `seq 1 40`; do (time sudo docker exec test_seccomp true &); done
MAX TIME : 2.009s

[Ioanna Alifieraki (joalif)]

# autopkgtest regression in EOAN

There is an autopkgtest regression in EOAN on ppc64el for systemd package.
The regression is caused by the 'upstream' test.
As it's shown here http://autopkgtest.ubuntu.com/packages/s/systemd/eoan/ppc64el this test
has always been failing since systemd version 240-6ubuntu5 and can be ignored.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libseccomp - 2.4.3-1ubuntu3.20.04.3

---------------
libseccomp (2.4.3-1ubuntu3.20.04.3) focal; urgency=medium

  * d/p/db-consolidate-some-of-the-code-which-adds-rules.patch
  * d/p/db-add-shadow-transactions.patch (LP: #1861177)
    Backport upstream patches to address performance regression introduced
    in libseccomp 2.4.

 -- Ioanna Alifieraki <email address hidden> Mon, 29 Jun 2020 13:35:12 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for libseccomp has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libseccomp - 2.4.3-1ubuntu3.18.04.3

---------------
libseccomp (2.4.3-1ubuntu3.18.04.3) bionic; urgency=medium

  * d/p/db-consolidate-some-of-the-code-which-adds-rules.patch
  * d/p/db-add-shadow-transactions.patch (LP: #1861177)
    Backport upstream patches to address performance regression introduced
    in libseccomp 2.4.

 -- Ioanna Alifieraki <email address hidden> Mon, 29 Jun 2020 13:52:22 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package libseccomp - 2.4.3-1ubuntu3.16.04.3

---------------
libseccomp (2.4.3-1ubuntu3.16.04.3) xenial; urgency=medium

  * d/p/db-consolidate-some-of-the-code-which-adds-rules.patch
  * d/p/db-add-shadow-transactions.patch (LP: #1861177)
    Backport upstream patches to address performance regression introduced
    in libseccomp 2.4.

 -- Ioanna Alifieraki <email address hidden> Mon, 29 Jun 2020 13:57:55 +0100

[Dariusz Gadomski (dgadomski)]

Marking Eoan as Won't fix due to EOL.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libseccomp - 2.4.3-1ubuntu3.19.10.3

---------------
libseccomp (2.4.3-1ubuntu3.19.10.3) eoan; urgency=medium

  * d/p/db-consolidate-some-of-the-code-which-adds-rules.patch
  * d/p/db-add-shadow-transactions.patch (LP: #1861177)
    Backport upstream patches to address performance regression introduced
    in libseccomp 2.4.

 -- Ioanna Alifieraki <email address hidden> Mon, 29 Jun 2020 13:47:58 +0100