new libseccomp 2.4 (in proposed) makes rebuilds need but not generate a dependency to 2.4

Bug #1830859 reported by Christian Ehrhardt  on 2019-05-29
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
libseccomp (Ubuntu)
Critical
Unassigned
qemu (Ubuntu)
Medium
Christian Ehrhardt 
Disco
Undecided
Unassigned

Bug Description

[Impact]

 * Just as of yesterday we have a new libseccomp in all releases.
   This brings some new code and features which might impact packages.
   For qemu being one of the few that is not so much a problem in many
   cases. Prior to Disco we didn't have code in qemu using the new
   libseccomp bits. In Eoan we can assume that libseccomp 2.4 (being
   release level) will be installed.
   But in Disco people might have installed and not updated libseccomp
   but install/upgrade to a qemu that was built against that new
   libseccomp. Due to that qemu will get a runtime dependency that is
   not picked up automatically - this would break all qemu execution
   being blocked by a failure to install seccomp filters.

 * To fix that we will add an explicit versioned dependency to qemu to
   make sure the rebuild will ensure that the right library version is
   available.

 * This is needed for qemu-system* which all depend on qemu-system-
   common so to avoid too much noise changing all qemu-system* packages
   the dependency it added there.

[Test Case]

 * ensure that the qemu install will ensure that libseccomp2 (>= 2.4.1)
   is also installed (not staying on 2.3)

 * Then install qemu and run
   $ qemu-system-x86_64 -enable-kvm -nographic -nodefaults -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny

   which should not trigger this (when fixed)
   error: internal error: process exited while connecting to monitor: 2019-05-28T17:10:17.121934Z qemu-system-x86_64: -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny: failed to install seccomp syscall filter in the kernel

[Regression Potential]

 * Well, the regressions might be triggered by pushing the new
   libseccomp, not by this dependency fixup. The security Team has
   checked and this seems to be the only affected code in the archive.
   So libseccomp was pushed to fix CVEs. I have thought if it is a
   regression that users of qemu will now be forced to use the newer
   libseccomp, but since it would be unusable without that is no
   important argument. And it is strongly recommended to upgrade for
   security reasons as well.

[Other Info]

 * n/a

---

It started with some of my usual KVM checks and found them failing on Disco with:
  error: internal error: process exited while connecting to monitor: 2019-05-28T17:10:17.121934Z qemu-system-x86_64: -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny: failed to install seccomp syscall filter in the kernel

I realized it works on X/B/C/E but not in a D container
It worked ~4 weeks ago.

This test can be simplified to one command:
$ qemu-system-x86_64 -enable-kvm -nographic -nodefaults -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny

With strace I found different behavior:
Good:
[pid 3487] 0.000000 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_TSYNC, NULL) = -1 EFAULT (Bad address) <0.000011>
[pid 3487] 0.003136 seccomp(SECCOMP_SET_MODE_STRICT, 1, NULL) = -1 EINVAL (Invalid argument) <0.000044>
[pid 3487] 0.000250 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_TSYNC, {len=68, filter=0x55f2b7a5e2d0}) = 0 <0.000251>

Bad:
[pid 484] 0.000000 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_TSYNC, NULL) = -1 EFAULT (Bad address) <0.000017>
[pid 484] 0.002680 seccomp(SECCOMP_GET_ACTION_AVAIL, 0, [SECCOMP_RET_KILL_PROCESS]) = 0 <0.000014>
[pid 484] 0.000088 seccomp(SECCOMP_SET_MODE_STRICT, 1, NULL) = -1 EINVAL (Invalid argument) <0.000013>

The difference seems to come from being built against seccomp as in Disco (2.3.3-3ubuntu2 = good) or Disco-proposed ( = bad).

The dependency on the built package stayed
 libseccomp2 (>= 2.3.0)
It did not pick up 2.4 via e.g. shlibs.

If I install libseccomp2 2.4 from -proposed the issue is gone.
Strace now looks like:
[pid 1788] 0.000000 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_TSYNC, NULL) = -1 EFAULT (Bad address) <0.000015>
[pid 1788] 0.004167 seccomp(SECCOMP_GET_ACTION_AVAIL, 0, [SECCOMP_RET_KILL_PROCESS]) = 0 <0.000013>
[pid 1788] 0.000098 seccomp(SECCOMP_SET_MODE_STRICT, 1, NULL) = -1 EINVAL (Invalid argument) <0.000013>
[pid 1788] 0.000054 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_TSYNC, NULL) = -1 EFAULT (Bad address) <0.000022>
[pid 1788] 0.000061 seccomp(SECCOMP_GET_ACTION_AVAIL, 0, [SECCOMP_RET_KILL_PROCESS]) = 0 <0.000017>
[pid 1788] 0.001477 seccomp(SECCOMP_SET_MODE_FILTER, SECCOMP_FILTER_FLAG_TSYNC, {len=68, filter=0x564de3852560}) = 0 <0.000288>

This is in all releases proposed to fix CVE-2019-9893
I was just ?lucky? to pick it up with my qemu build in the PPA that has proposed enabled.

Now there might be some toolchain detail to this as well.
As I built qemu for B/C as well and they built against the proposed 2.4 versions as well.
But in B/C things work with new qemu builds and old libseccomp2 installed.
Eoan on the other hand works by having libseccomp2 2.4.1-0ubuntu0.19.10.3 already migrated.

But that "luck" of B/C working might be specific to Qemus code, other rebuilds against the new libseccomp2 might fail as well in B/C and further. Shlibs detection is based on these symbols but my new qemu build is not calling these so it dependency stays at 2.3.

Until a simpler testcase is found, I have set up two new PPAs:
https://launchpad.net/~paelzer/+archive/ubuntu/bug-1830859-with-proposed-seccomp
https://launchpad.net/~paelzer/+archive/ubuntu/bug-1830859-without-proposed-seccomp

In comment #5 I debugged the case and found the new "dependency"
As a summary:
- the libseccomp-dev 2.4 headers being installed at build time
- code can detect now the availability of SCMP_ACT_KILL_PROCESS
- code might decide to use SCMP_ACT_KILL_PROCESS
- if code runs without libseccomp2 2.4 installed it breaks

### some related Build logs ###
Cosmic rebuild against 2.4, not affected by the issue
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3729/+build/16858331/+files/buildlog_ubuntu-cosmic-amd64.qemu_1%3A2.12+dfsg-3ubuntu8.9~ppa1_BUILDING.txt.gz

Disco rebuild against 2.4, affected by the issue
https://launchpadlibrarian.net/425689546/buildlog_ubuntu-disco-amd64.qemu_1%3A3.1+dfsg-2ubuntu3.2~ppa1_BUILDING.txt.gz

Disco build against 2.3 (working)
https://launchpadlibrarian.net/422884897/buildlog_ubuntu-disco-amd64.qemu_1%3A3.1+dfsg-2ubuntu3.1_BUILDING.txt.gz

Task for now is:
- Critical (for the chance of breaking a potentially arbitrary amount of SRUs that were built against it in proposed)
- Assigned to ubuntu-security as they are driving the libseccomp update

We need to understand it more to then re-triage severity and potential actions.

Changed in ubuntu:
importance: Undecided → Critical
assignee: nobody → Ubuntu Security Team (ubuntu-security)
no longer affects: ubuntu
Changed in libseccomp (Ubuntu):
assignee: nobody → Ubuntu Security Team (ubuntu-security)
importance: Undecided → Critical

Until a simpler testcase is found, the qemu in PPA [1] built for Disco will trigger it.
=> https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3729/+packages

description: updated

To be clear - this will likely resolve over time when people upgrade to seccomp 2.4.
But right now SRUs could release things built in -proposed against it and then end up broken until 2.4 is released as well.
Furthermore without a strict dependency it might stay broken as nothing enforces 2.4 to be installed.

I have set up two new PPAs
https://launchpad.net/~paelzer/+archive/ubuntu/bug-1830859-with-proposed-seccomp
https://launchpad.net/~paelzer/+archive/ubuntu/bug-1830859-without-proposed-seccomp

They both build amd64 qemu with/without proposed for Disco.
Including debug symbols for further debugging.

This decouples this from my other PPA to allow me going on there with a temporary workaround.

description: updated
Download full text (3.6 KiB)

Two Disco containers (one per PPA)

$ sudo add-apt-repository ppa:paelzer/bug-1830859-with-proposed-seccomp
or
$ sudo add-apt-repository ppa:paelzer/bug-1830859-without-proposed-seccomp
# add main/debug and enable the deb-src in the PPA sources

$ sudo apt install gdb qemu-system-x86 qemu-system-x86-dbgsym dpkg-dev
$ apt source qemu
$ cd qemu-3.1+dfsg/

break on qemu_seccomp

Hrm ??
equal qemu source
Good:
Breakpoint 1 at 0x4d63f4: file ./qemu-seccomp.c, line 293.
Bad:
Breakpoint 1 at 0x4d63f4: qemu_seccomp. (2 location
(gdb) info b
Num Type Disp Enb Address What
1 breakpoint keep y <MULTIPLE>
1.1 y 0x00000000004d63f4 in qemu_seccomp at ./qemu-seccomp.c:293
1.2 y 0x00000000004d656e in qemu_seccomp at ./qemu-seccomp.c:244

Now the second hit isn't an actual qemu_seccomp call, but it is very close to the error that we get reported.

(gdb) l qemu-seccomp.c:293
288
289 #if defined(SECCOMP_FILTER_FLAG_TSYNC)
290 int check;
291
292 /* check host TSYNC capability, it returns errno == ENOSYS if unavailable */
293 check = qemu_seccomp(SECCOMP_SET_MODE_FILTER,
294 SECCOMP_FILTER_FLAG_TSYNC, NULL);
295 if (check < 0 && errno == EFAULT) {
296 add = true;
297 }
(gdb) l qemu-seccomp.c:244
239 error_setg(errp, "invalid argument for resourcecontrol");
240 return -1;
241 }
242 }
243
244 if (seccomp_start(seccomp_opts) < 0) {
245 error_setg(errp, "failed to install seccomp syscall filter "
246 "in the kernel");
247 return -1;
248 }

Well maybe with seccomp 2.4 development headers something changed and this now gets inlined?
That is not an error, but suspicious.

qemu_seccomp should ALWAYS be inlined

static inline __attribute__((unused)) int
qemu_seccomp(unsigned int operation, unsigned int flags, void *args)

The one at line 293 is at seccomp_register and depends on #if defined(SECCOMP_FILTER_FLAG_TSYNC).
This one is active in both cases.

But the one at line 244 is part of qemu_seccomp_get_kill_action
That has some checks as well being:
#if defined(SECCOMP_GET_ACTION_AVAIL) && \
    defined(SCMP_ACT_KILL_PROCESS) && \
    defined(SECCOMP_RET_KILL_PROCESS)

There is no other way to "avoid" the path of
  parse_sandbox -> seccomp_start -> qemu_seccomp_get_kill_action -> qemu_seccomp
in terms of precompiler.
Both have CONFIG_SECCOMP set, so it might really come down to the three checks above.

Reading the commit [1] that added this to qemu seems to match the current theories.

Lets see what happens when it tries to use things buildt with 2.4 but without 2.4 being installed at runtime.

The returned value for action is matching expected /usr/include/seccomp.h numbers:
- good case => 196608 ==> SCMP_ACT_TRAP = 0x00030000U
- bad case => 2147483648 ==> SCMP_ACT_KILL_PROCESS = 0x80000000U

As a summary:
- the libseccomp-dev 2.4 headers being installed at build time
- code can detect now the availa...

Read more...

description: updated

[13:09] <mdeslaur> cpaelzer: oh, we knew about that one :(
[13:09] <cpaelzer> well
[13:09] <cpaelzer> then had you a plan how we should resolve that
[13:10] <cpaelzer> either in general (magic great code I haven't thought of) or in particular (how I should resolve it for qemu) ?
[13:10] <cpaelzer> and good morning mdeslaur sorry to start with such issues :-/
[13:10] <mdeslaur> cpaelzer: good morning! :)
[13:11] <-- lborda-eod (lborda@173-246-24-26.qc.cable.ebox.net) has left this server (Quit: Ex-Chat).
[13:11] <mdeslaur> cpaelzer: the plan is to publish the libseccomp update ;)
[13:11] --> lborda (lborda@173-246-24-26.qc.cable.ebox.net) has joined this channel.
[13:11] <mdeslaur> cpaelzer: we searched the archive, and qemu was the only problematic package that used those new apis
[13:12] <mdeslaur> jdstrand: are we ready to publish the seccomp updates?
[13:12] <cpaelzer> mdeslaur: ok then should I add an explicit dependency for Disco
[13:12] <cpaelzer> only there the "transition" might occur
[13:13] <cpaelzer> that we rebuild with the new version around but the old might be installed
[13:13] <jjohansen> hey mdeslaur
[13:13] <cpaelzer> I'm prepping an SRU anyway and that would avoid the "rebuilt to 2.4 but not installed"
[13:13] <cpaelzer> if qemu really is the only one I'm ok to carry this
[13:13] <mdeslaur> cpaelzer: yeah, you can add an explicit dependency, or you can wait until we publish seccomp...we may publish it today or monday
[13:13] <cpaelzer> and in Eoan things are fine as that will release with >=2.4
[13:14] <cpaelzer> mdeslaur: well, my point is even if you publish it there are no guarantees
[13:14] <cpaelzer> people might update qemu but not libseccomp
[13:14] <cpaelzer> and they would be screwed
[13:14] <cpaelzer> so the first rebuild of qemu in disco (=now) will need to add the dependency
[13:14] <cpaelzer> mdeslaur: ^^
[13:15] <mdeslaur> yes, selective installing of security updates is an untested and probably broken configuration
[13:15] <mdeslaur> if you add the explicit dependency, you'll need to wait until we do publish seccomp too
[13:15] <cpaelzer> yes, but I'll need a few days anyway
[13:15] <cpaelzer> and you seem close
[13:16] <mdeslaur> ok
[13:16] <cpaelzer> and for testing interim wise I can throw a no change rebuild of libseccomp 2.4 in my ppa
[13:16] <mdeslaur> I'll try and remember to add the explicit dependency to the next round of qemu updates
[13:16] <cpaelzer> Hmm, you have some in the pipe as well?
[13:16] <mdeslaur> I don't currently, no
[13:16] <cpaelzer> I have
[13:16] <cpaelzer> I'll add them right away
[13:17] <cpaelzer> if testing goes well those should hit -unapproved early next week
[13:17] <mdeslaur> sorry our timezones don't overlap and you had to spend time debugging this
[13:17] <mdeslaur> we spotted this issue when we prepared the emergency qemu update last week

TL;DR
we will make is a qemu fix in Disco, by adding an explicit dependency.
All others don't need the change.

Changed in libseccomp (Ubuntu):
status: New → Won't Fix
Changed in qemu (Ubuntu):
status: New → Triaged
assignee: nobody → Christian Ehrhardt  (paelzer)
Changed in libseccomp (Ubuntu):
assignee: Ubuntu Security Team (ubuntu-security) → nobody
Changed in qemu (Ubuntu):
importance: Undecided → Medium
Jamie Strandboge (jdstrand) wrote :

FYI, libseccomp is now published: https://usn.ubuntu.com/4001-1/

Thanks Jamie,
as discussed an explicit dependency as fix will be part of the next SRU upload to qemu to avoid anybody running into trouble due to this.

description: updated

Tested on the PPA, working as expected (after moved to qemu-system-*packacges)
The reason to no more use qemu-system-common is that this only was a recommends which would not enforce the update.
Even qemu-system-data which is a depends would be wrong as people could install that without the actual binaries and they would not need the new libseccomp.

Lets take the bitter pill and add them exactly where needed.
I analyzed binaries - qemu-user and others don't need it.

Checked on the PPA:

root@d-wily:~# apt install libseccomp2=2.3.3-3ubuntu2
Reading package lists... Done
Building dependency tree
Reading state information... Done
[...]
The following packages will be REMOVED:
  qemu-kvm qemu-system-x86
The following packages will be DOWNGRADED:
  libseccomp2
0 upgraded, 0 newly installed, 1 downgraded, 2 to remove and 0 not upgraded.

... and similar.
So the dependency works and makes sure a recent libseccomp2 will have to go along and by that prevent the error.

Uploaded to Disco-unapproved.

Hello Christian, or anyone else affected,

Accepted qemu into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:3.1+dfsg-2ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in qemu (Ubuntu Disco):
status: New → Fix Committed
tags: added: verification-needed verification-needed-disco

On this rebuild not only did it pick up the new libseccomp for build
   libseccomp2 amd64 2.4.1-0ubuntu0.19.04.3
but also has the dependency to libseccomp2 (>= 2.4.1) to ensure it has to be upgraded.

Seen in e.g.: https://launchpadlibrarian.net/427212564/buildlog_ubuntu-disco-amd64.qemu_1%3A3.1+dfsg-2ubuntu3.2_BUILDING.txt.gz

In a container I tried to fail-install (keep old libseccomp) but the dependency makes sure this won't happen

Works fine, setting verified

tags: added: verification-done verification-done-disco
removed: verification-needed verification-needed-disco
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:3.1+dfsg-2ubuntu3.2

---------------
qemu (1:3.1+dfsg-2ubuntu3.2) disco; urgency=medium

  * d/p/ubuntu/define-ubuntu-machine-types.patch: fix wily machine type being
    broken since 2.11 due to 2.3/2.4 version mismatch in its definition to
    fix migrations from old machines (LP: #1829868).
  * d/p/ubuntu/lp-1830704-s390x-cpumodel-ignore-csske-for-expansion.patch
    toleration for future machines (LP: #1830704
  * d/control-in, d/control: add versioned dependencies to libseccomp 2.4 as
    any rebuild against 2.4 as it is in proposed right now will otherwise
    crash (LP: #1830859).

 -- Christian Ehrhardt <email address hidden> Tue, 28 May 2019 10:52:47 +0200

Changed in qemu (Ubuntu Disco):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

As Eoan releases with >=2.4 this isn't a problem for Eoan, closing the task.

Changed in qemu (Ubuntu):
status: Triaged → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers