Can't load seccomp filter

[Nino Škopac]

After migrating from Ubuntu 20 amd64 to aarch64 I started experiencing "can't load seccomp filter" when doing `apt update && apt upgrade` and "Kernel refuses to turn on BPF filters" when using Puppeteer.

I wrote about it more extensively here: https://stackoverflow.com/questions/69892137/after-a-few-days-i-can-no-longer-start-puppeteer-until-i-restart-the-server

lsb_release -rd
---------------
Description: Ubuntu 20.04.3 LTS
Release: 20.04

apt-cache policy seccomp
---------------
seccomp:
  Installed: (none)
  Candidate: 2.5.1-1ubuntu1~20.04.2
  Version table:
     2.5.1-1ubuntu1~20.04.2 500
        500 http://us-east-1.ec2.ports.ubuntu.com/ubuntu-ports focal-updates/main arm64 Packages
        500 http://ports.ubuntu.com/ubuntu-ports focal-security/main arm64 Packages
     2.4.3-1ubuntu1 500
        500 http://us-east-1.ec2.ports.ubuntu.com/ubuntu-ports focal/main arm64 Packages

[Lucas Kanashiro (lucaskanashiro)]

Thanks for taking the time to report this bug and trying to make Ubuntu better.

I am not familiar with Puppeteer but since you think this might be related to kernel, could you please provide any system logs so we can understand what is happening? In a regular arm64 VM I am able to run "apt update && apt upgrade" without issues (FWIW I am not running it on AWS).

I am setting the status of this bug to Incomplete, once you provide more information set it back to New and we will take a look at it again.

[Nino Škopac (theonin)]

Dear Lucas,

I've mentioned in the SO post that I linked that the nature of the issue is intermittent.

I would be happy to provide additional logs next time I encounter the issue.

Could you please advise on how do I collect additional logs?

Regards,
Nino

[Paride Legovini (paride)]

Hi Nino, I found a RedHat bug [1] that describes a similar situation (affected is dhclient on ppc64le, but the error is the same).

The suggested workaround is setting

  net.core.bpf_jit_limit = 262144000

via sysctl. I checked and on amd64 I find

  net.core.bpf_jit_limit = 264241152

while on an arm64 machine that's:

  net.core.bpf_jit_limit = 33554432

(both machines are running Bionic), so it looks arch-dependent indeed. Could you please try bumping that value, and check if the issue goes away?

If it does go away we can ask the kernel team what they think about bumping it by default.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1647947

[Nino Škopac (theonin)]

Dear Paride,

Thank you so much!!!!

I'll wait for the issue to resurface and *then* bump the value in order to have conclusive evidence.

How should I do it?
echo "net.core.bpf_jit_limit = 262144000" >> /etc/sysctl.conf (as sudo)

Will that work without restart? Because the problem goes away after I restart.

Regards,
Nino

[Paride Legovini (paride)]

If you want to change the value without rebooting that's:

  sudo sysctl net.core.bpf_jit_limit=264241152

Add it to /etc/sysctl.conf if you want to set it automatically at boot, but it's more meaningful to test as you said: wait for the issue to happen and then change the value while the system is running.

[Nino Škopac (theonin)]

I confirmed the solution, it worked.

Also, I tried to submit the patch against the linux kernel (and/or Seccomp-BPF subtree) following the official kernel guide (https://www.kernel.org/doc/html/latest/process/submitting-patches.html) but I couldn't even find the 33554432 value by doing:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
grep 33554432 -R linux

Got this:
linux/tools/power/x86/turbostat/turbostat.c:unsigned int irtl_time_units[] = { 1, 32, 1024, 32768, 1048576, 33554432, 0, 0 };
linux/drivers/idle/intel_idle.c: 1, 32, 1024, 32768, 1048576, 33554432, 0, 0
linux/drivers/pci/pcie/aspm.c: } else if (threshold_ns < 33554432) {
linux/drivers/atm/idt77252_tables.h:/* 33554432.00 => 255 */ 0xff, /* => 352768.00 */
linux/drivers/atm/idt77252_tables.h:/* 335544320.00 => 255 */ 0xff, /* => 352768.00 */
linux/drivers/atm/idt77252_tables.h:/* 3355443200.00 => 255 */ 0xff, /* => 352768.00 */
linux/drivers/platform/x86/intel/pmc/core.c: * | 5 | 33554432 |
linux/drivers/iio/frequency/adf4371.c:#define ADF4371_MODULUS1 33554432ULL
linux/arch/arm/boot/dts/imx51-apf51.dts: clock-frequency = <33554432>;
linux/arch/arm/kernel/insn.c: if (unlikely(offset < -33554432 || offset > 33554428)) {
linux/mm/slab_common.c: INIT_KMALLOC_INFO(33554432, 32M)
linux/sound/soc/codecs/rt1305.c: r0ohm = (rhl*10) / 33554432;
linux/sound/soc/codecs/rt1305.c: r0ohm = (rhl*10) / 33554432;
linux/sound/soc/codecs/twl4030.c: 8388608, 16777216, 33554432, 67108864};

None of which is an arm-specific constant for Seccomp-BPF.

I tried.

Once again, thank you for your help Paride Legovini, I would have never found this on my own.

I gotta say, I'm very curious as to why this happens.
What does the bpf_jit_limit value actually mean?
I what what's BPF, I know what's JIT, I just don't know how they mix.

[Paride Legovini (paride)]

Hi, glad to know it worked. There is some heuristics behind the default bpf_jit_limit [1], it isn't a simple hardcoded value. We may discuss bumping the default in Ubuntu, but I don't think that's a good idea: the in-kernel heuristics has certainly been well thought, and just bumping the number is likely to have other unintended consequences.

My take here is: your setup needs tuning, and that's what those config knobs are for. Note that it's better to add a config file under /etc/sysctl.d rather than modifying the default /etc/sysctl.conf.

Let me know if this makes sense for you. I'm leaving this bug marked Incomplete for now.

[1] https://github.com/torvalds/linux/blob/8efd0d9c316af470377894a6a0f9ff63ce18c177/kernel/bpf/core.c#L826

[Nino Škopac (theonin)]

My hunch is that the Puppeteer instances (which are Chromium instances) saturate the allocated memory limit for BPF, because they restart quite a lot, so maybe each time a new instance starts, it makes BPF allocate more memory, until it's full.

But I have no idea:
- how does JIT memory limit correlate to this? Isn't BPF pruned?
- about how BPF works internally

I wonder how could I tune my setup.

Thanks for letting me know where is the constant computed (FYI I don't understand these heuristics at all).