Disabling bpf() syscall on kernel lockdown break apps when secure boot is on

Bug #1863234 reported by Quentin Monnet
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Undecided
Unassigned
Disco
Medium
Tyler Hicks
Eoan
Medium
Tyler Hicks

Bug Description

[Impact]

The bpf(2) system call is completely blocked in Disco and Eoan when Secure Boot is enabled due to overly restrictive Lockdown policies. This makes it so that all bpf related tools are not usable on those releases.

[Test Case]

Set up test BPF programs:

$ cat hello.bt
BEGIN { printf("hello\n"); exit(); }
$ cat kprobe.bt
kprobe:do_nanosleep { printf("task sleeping...\n"); exit(); }
$ cat open.bt
tracepoint:syscalls:sys_enter_openat {
  printf("filename: [%s]; flags: [%d]\n",
         str(args->filename), args->flags);
}

Disable Secure Boot:

$ sudo mokutil --disable-validation
...
$ sudo reboot

Ensure that hello.bt can run:

$ sudo bpftrace hello.bt
Attaching 1 probe...
hello

Ensure that a BPF program triggered by a kprobe works (run `sleep 1` in another terminal):
$ sudo bpftrace kprobe.bt
Attaching 1 probe...
task sleeping...

Ensure that a BPF program triggered by a tracepoint can access the filename and flags of openat(2):
$ sudo bpftrace open.bt
Attaching 1 probe...
filename: [/proc/2317/cmdline]; flags: [0]
filename: [/dev/iio:device1]; flags: [2048]
...

Enable Secure Boot

$ sudo mokutil --enable-validation
...
$ sudo reboot

Ensure that a basic BPF program can run:

$ sudo bpftrace hello.bt
Attaching 1 probe...
hello

Ensure that a BPF program triggered by a kprobe is blocked (kprobes aren't allowed under Secure Boot):
$ sudo bpftrace kprobe.bt
Attaching 1 probe...
cannot attach kprobe, Operation not permitted
Error attaching probe: 'kprobe:do_nanosleep'

You should see the following kernel message logged:

Lockdown: bpftrace: Use of kprobes is restricted; see man kernel_lockdown.7

Ensure that a BPF program triggered by a tracepoint can NOT access the filename and flags of openat(2) (all filenames should be empty and all flags should be 0):
$ sudo bpftrace open.bt
Attaching 1 probe...
filename: []; flags: [0]
filename: []; flags: [0]
...

You should see the following kernel message logged:

Lockdown: iio-sensor-prox: BPF is restricted; see man kernel_lockdown.7

[Regression Potential]

Low. This is opening up the use of bpf(2) while under Lockdown. There should be no new restrictions put in place.

[Original Report]

In disco and eoan, lockdown is automatically enforced when secure boot is on [0]. Because lockdown was not in the mailine kernel at the time, some disrto-specific patches were added to the kernel, including one that drastically restricts BPF usage by completely disabling the use of the `bpf()` system call when lockdown is on [1].

A consequence of that decision is that no application relying on eBPF can run on 19.04/19.10, unless secure boot / lockdown is disabled. For example, Cilium (cilium.io) strongly relies on BPF programs to implement its datapath and securing network connectivity between containers. Other applications like Suricata or Sysdig also rely on BPF to some extent. None of which will work by default on a EFI machine with secure boot activated.

If I understand correctly, kernel 5.4 (to be used in focal) will have a different, lighter restricton (comming from mainline Linux kernel) [2], so `bpf()` for networking use cases should mostly work on 20.04. Is my understanding correct? If so, could this patch be backported to 19.10 (and 19.04, if still supported) instead of completely disabling the syscall on lockdown?

Links:
[0] https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/disco/commit/?id=d0db99473fc3bb8a5d03f99ed454ac7ca5e7d517
[1] https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/disco/commit/?id=2a68c65abae66d28e2acb3245cb156ae2ea6eb1d
[2] https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/focal/commit/?id=9d1f8be5cf42b497a3bddf1d523f2bb142e9318c

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1863234

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Quentin Monnet (qmonnet) wrote :

Not adding kernel logs but changing to 'Confirmed'.

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Tyler Hicks (tyhicks) wrote :

Hi Quentin - Thanks for the bug report! I do think that relaxing the eBPF restrictions in Eoan and Disco would be acceptable for Secure Boot purposes.

Changed in linux (Ubuntu Disco):
status: New → Triaged
Changed in linux (Ubuntu Eoan):
status: New → Triaged
Changed in linux (Ubuntu Disco):
importance: Undecided → Medium
Changed in linux (Ubuntu Eoan):
importance: Undecided → Medium
Changed in linux (Ubuntu):
status: Confirmed → Invalid
Tyler Hicks (tyhicks)
Changed in linux (Ubuntu Disco):
status: Triaged → In Progress
Changed in linux (Ubuntu Eoan):
status: Triaged → In Progress
Changed in linux (Ubuntu Disco):
assignee: nobody → Tyler Hicks (tyhicks)
Changed in linux (Ubuntu Eoan):
assignee: nobody → Tyler Hicks (tyhicks)
Tyler Hicks (tyhicks)
description: updated
Tyler Hicks (tyhicks)
description: updated
Revision history for this message
Tyler Hicks (tyhicks) wrote :
Revision history for this message
Brendan Gregg (brendangregg) wrote :

The relaxed BPF restrictions still break BPF tracing and other things, making Ubuntu no longer meet the debugability requirements for an enterprise OS.

Lockdown should not be enabled by default. It needs to be opt-in, not opt-out.

Tyler -- please fix Ubuntu.

Revision history for this message
Brendan Gregg (brendangregg) wrote :

This change also prevents BPF security programs from running (like those we use at Netflix) making Ubuntu less secure.

In case I'm not being clear enough: this is the worst change I've ever seen in operating systems.

Some people want lockdown? Let them opt in.

Revision history for this message
Tyler Hicks (tyhicks) wrote :

Hi Brendan - What you're asking for is very different than the intent behind this bug report. It'll be best if you open a new bug report.

Changed in linux (Ubuntu Eoan):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Disco):
status: In Progress → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-eoan' to 'verification-done-eoan'. If the problem still exists, change the tag 'verification-needed-eoan' to 'verification-failed-eoan'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-eoan
Revision history for this message
Quentin Monnet (qmonnet) wrote :

Tested kernel 5.3.0-43-generic from -proposed, on eoan with Secure Boot/Lockdown enabled. Running 'sudo bpftool prog' works and lists BPF programs loaded on the system, via the bpf() syscall. Same test on 5.3.0-42-generic would fail with -EPERM.

So the fix works well, and we can now use bpf() even with Lockdown, thanks! I'll update the verification tag. This is definitely an improvement, although the resolution here will not address Brendan's concerns for tracing.

tags: added: verification-done-eoan
removed: verification-needed-eoan
Revision history for this message
Stefan Bader (smb) wrote :

This was reverted due to bug #1868626

Changed in linux (Ubuntu Eoan):
status: Fix Committed → Triaged
Steve Langasek (vorlon)
Changed in linux (Ubuntu Disco):
status: Fix Committed → Won't Fix
Revision history for this message
Brian Murray (brian-murray) wrote :

The Eoan Ermine has reached end of life, so this bug will not be fixed for that release

Changed in linux (Ubuntu Eoan):
status: Triaged → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers