[UBUNTU 18.04] BPF programs fail on Ubuntu s390x

[bugproxy]

[Impact]
Some bpf programs will fail to execute on s390x, returning EFAULT when they should be able to read user memory.

[Test case]
apt-get source linux
mkdir -p /usr/lib/perf/
cp -a linux-5.4.0/tools/perf/include /usr/lib/perf/
probe_read=$(grep -w probe_read /usr/lib/perf/include/bpf/bpf.h)
probe_read_user=${probe_read//read/read_user}
sed -i "/probe_read)/i$probe_read_user" /usr/lib/perf/include/bpf/bpf.h
probe_read_user_str=${probe_read//read/read_user_str}
sed -i "/probe_read)/i$probe_read_user_str" /usr/lib/perf/include/bpf/bpf.h

ed - linux-5.4.0/tools/perf/examples/bpf/augmented_raw_syscalls.c << EOF
100c
int string_len = probe_read_user_str(&augmented_arg->value, arg_len, arg);
.
w
EOF
perf trace -eopenat,augmented_raw_syscalls.c cat /etc/passwd > /dev/null

You should see:
     0.332 ( 0.002 ms): cat/3223 openat(dfd: CWD, filename: "/etc/passwd") = 3
instead of
     0.334 ( 0.003 ms): cat/3739 openat(dfd: CWD, filename: "") = 3

[Potential regressions]
One potential regression is that unprivileged code can be able to exploit the changes to read or write kernel memory.

--------------------------------------------------------------------------

We need to run BPF filters to analyse and monitor network traffic. The BPF filters are created by skydive (http://skydive.network). Currently skydive fails to install BPF filters on s390x (using Ubuntu 18.04 currently, soon moving to Ubuntu 20.04).

Because of these failures, we decided to try the BPF samples that come with the kernel first. These samples also fail on s390x while they work fine on Intel.

shense@boe-build:~/bionic/samples/bpf$ uname -a
Linux boe-build 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:40:36 UTC 2020 s390x s390x s390x GNU/Linux

Example instructions:

sudo apt install -y dpkg-dev clang llvm libelf-dev
sudo apt-get source linux-image-unsigned-$(uname -r)
cd linux-4.15.0/
make headers_install
make samples/bpf/

Errors:

shense@boe-build:~/bionic/samples/bpf$ sudo ./test_map_in_map
[sudo] password for shense:
invalid relo for insn[4].code 0x85
bpf_load_program() err=22
0: (bf) r7 = r1
1: (b7) r1 = 0
2: (63) *(u32 *)(r10 -32) = r1
3: (bf) r1 = r7
4: (85) call unknown#-1
BPF_CALL uses reserved fields
0: (bf) r7 = r1
1: (b7) r1 = 0
2: (63) *(u32 *)(r10 -32) = r1
3: (bf) r1 = r7
4: (85) call unknown#-1
BPF_CALL uses reserved fields

shense@boe-build:~/bionic/samples/bpf$ sudo ./map_perf_test
invalid relo for insn[22].code 0x85
bpf_load_program() err=22
0: (bf) r7 = r1
1: (18) r1 = 0x207265743a25646e
3: (7b) *(u64 *)(r10 -16) = r1
4: (18) r1 = 0x705f616c6c6f632e
6: (7b) *(u64 *)(r10 -24) = r1
7: (18) r1 = 0x5f6c72755f686d61
9: (7b) *(u64 *)(r10 -32) = r1
10: (18) r1 = 0x7420737472657373
12: (7b) *(u64 *)(r10 -40) = r1
13: (18) r1 = 0x4661696c65642061
15: (7b) *(u64 *)(r10 -48) = r1
16: (b7) r1 = 0
17: (73) *(u8 *)(r10 -8) = r1
18: (b7) r2 = 1
19: (7b) *(u64 *)(r10 -72) = r2
20: (63) *(u32 *)(r10 -76) = r1
21: (bf) r1 = r7
22: (85) call unknown#-1
BPF_CALL uses reserved fields
0: (bf) r7 = r1
1: (18) r1 = 0x207265743a25646e
3: (7b) *(u64 *)(r10 -16) = r1
4: (18) r1 = 0x705f616c6c6f632e
6: (7b) *(u64 *)(r10 -24) = r1
7: (18) r1 = 0x5f6c72755f686d61
9: (7b) *(u64 *)(r10 -32) = r1
10: (18) r1 = 0x7420737472657373
12: (7b) *(u64 *)(r10 -40) = r1
13: (18) r1 = 0x4661696c65642061
15: (7b) *(u64 *)(r10 -48) = r1
16: (b7) r1 = 0
17: (73) *(u8 *)(r10 -8) = r1
18: (b7) r2 = 1
19: (7b) *(u64 *)(r10 -72) = r2
20: (63) *(u32 *)(r10 -76) = r1
21: (bf) r1 = r7
22: (85) call unknown#-1
BPF_CALL uses reserved fields

[Thadeu Lima de Souza Cascardo (cascardo)]

After some investigation, I found out that failures caused by "invalid relo for insn[4].code 0x85" are due to a small typo in some headers.

That has caused LLVM to emit relocations (thinking those missing macro calls were external function calls) that are not supported by the loader. This is all in userspace, so no real kernel bugs here. We can include the typo fixup in the bionic tree, though.

That led to a new failure, caused by faults when trying to read user memory from the BPF program. I will investigate further.

Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-07-28 08:05 EDT-------
Interesting, this might explain why the relocation issue did not trigger for me.

In the meantime I posted fixes to other bpf/samples issues: https://<email address hidden>/

[Thadeu Lima de Souza Cascardo (cascardo)]

Yep, and the commit that first fixes up these probe_read vs probe_read_kernel/probe_read_user problems is:

6ae08ae3dea2cfa03dd3665a3c8475c2d429ef47 "bpf: Add probe_read_{user, kernel} and probe_read_{user, kernel}_str helpers"

So, these helpers are not available until 5.5. We may consider this a bug, but BPF code will still need to be changed to use the new helpers, otherwise they will keep failing on s390x.

Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-08-04 18:45 EDT-------
Hi,

I fixed the skydive header as follows:

--- a/ebpf/defs.h
+++ b/ebpf/defs.h
@@ -56,7 +56,7 @@ static __inline int discret_memcmp(char *s1, char *s2, int n)
/* helper marcro to define a map, socket, kprobe section in the
* eBPF elf file.
*/
-#define MAP(NAME) struct bpf_map_def __section("maps/" #NAME) NAME =
+#define MAP(NAME) struct bpf_map_def __section("maps") NAME =
#define SOCKET(NAME) __section("socket_" #NAME)
#define LICENSE __section("license")
and also tried to cherry-pick some of the older kernel patches I wrote, e.g.:

https://git.kernel.org/pub/scm/linux/kernel/git/s390/linux.git/commit/?h=features&id=88aa8939c96781089e5ace3492d818074c5c6fe9
https://git.kernel.org/pub/scm/linux/kernel/git/s390/linux.git/commit/?h=features&id=05a68e892e89c97df6650cd8cc55058002657cbc
https://git.kernel.org/pub/scm/linux/kernel/git/s390/linux.git/commit/?h=features&id=3f161e0ae863a0456d00e5a6c9c81098c62ab7fe

- the new ones I linked here fix samples and aren't relevant for skydive - but then I found out that quite some common code is missing as well. For example, on 20.04 LLVM 10 is default. It contains the following commit:

commit fbb64aa69835c8e3e9efe0afc8a73058b5a0fb3c # llvmorg-10.0.0
Author: Yonghong Song <email address hidden>
Date: Tue Dec 17 16:24:23 2019 -0800

[BPF] extend BTF_KIND_FUNC to cover global, static and extern funcs

which makes clang produce non-backwards-compatible BTF. This leads to:

[14] FUNC bpf_flow_table type_id=12 vlen != 0
libbpf: Error loading .BTF into kernel: -22.

when loading. It's not that BPF devs did not think about this: in case kernel cannot handle the new BTF_KIND_FUNC bits, they are supposed to be sanitized by libbpf. However, Ubuntu 20.04 does not ship the newest libbpf:

# /usr/sbin/bpftool --version
/usr/lib/linux-tools/5.4.0-42-generic/bpftool v5.4.44

whereas the commit that introduced sanitization is in v5.6:

commit 2d3eb67f64ec317bbfe340cfcc8325d40a6ff317 # v5.6
Author: Alexei Starovoitov <email address hidden>
Date: Thu Jan 9 22:41:19 2020 -0800

libbpf: Sanitize global functions

Since this now turns into quite an adventure w.r.t. backporting, can't we try an easier path first? Namely: use clang-8, which is also available in Ubuntu 20.04, but doesn't include any fancy stuff!

skydive# CLANG=clang-8 GOPATH=$HOME/go make -C ebpf
skydive# bpftool prog load ebpf/flow.o /sys/fs/bpf/flow ; echo $?
0
# bpftool prog load ebpf/flow-gre.o /sys/fs/bpf/flow-gre ; echo $?
0

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-08-14 06:51 EDT-------
@Canonical: what is the current expectation on solving this issue. This one it timely critical for our upcoming offering. A comment from Canonicals side would be appreciated.. Many thx in advance

[Thadeu Lima de Souza Cascardo (cascardo)]

Hi.

So, we clarified that some of the problems were not caused by the kernel at all. The only thing missing is support to probe_read_kernel and probe_read_user on 4.15 kernels. Is that needed for the offering?

If I didn't get it right, I am sorry. So, in order for us to meet your expectations, I'll try to understand better what are the needs here:

1) Do you want support on 4.15 kernel on Ubuntu 18.04, right? Or is the offering based on Ubuntu 20.04?

2) Do you expect to be able to use LLVM/clang 10? Or is LLVM/clang 8 sufficient for your needs? The bug has only been open for the linux package, so we would need to get folks behind LLVM involved, or try to fix this on the kernel somehow, which might delay things a little. Being able to use LLVM/clang 8, it seems, would be a best fit to a short deadline.

3) Building and running samples/bpf/ is not something we test. And though it might help us catch regressions, I would suppose it's not a requirement for the offering. If I am wrong in that supposition, are you relying on the source package for something? Or is anything shipped in binary packages incorrect (like that s930x typo) preventing things from building correctly?

Right now, the best course we have, in case what you need is the probe functions on 4.15 is to get this available on a kernel in -proposed in 3 weeks, and in -updates in 6 weeks. Would that work?

Thank you.
Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-08-18 06:26 EDT-------
@Cascado:

IBM's response to your comment suggestion: wer:

1) the offering will be based on Ubuntu 20.04

2) it is okay for us to use clang-8
(actually the userspace code runs in a Ubuntu 18.04
container including the clang-8 package)

3) we will rerun the workload to check if this combination works fine.

Does that answer your requesting and let you proceed.
Many thanks in advance

[Thadeu Lima de Souza Cascardo (cascardo)]

Hi.

So if I understand correctly, this will be 18.04 userspace on top of a 20.04 kernel, that is, 5.4. I noticed 5.4 also lacks probe_read_user. I will keep working on a backport of this to 5.4 kernel, then. Let me know if there is anything else missing here.

Thanks.
Cascardo.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-09-01 03:25 EDT-------
und BPF ist derzeit nicht mehr dringend, da arbeiten wir jetzt upstream, und werden zu einem sp?teren zeitpunt wiederkommen falls es ein target date gibt, aber das schreib ich dir noch in den LP

Status update from IBM : Currently this feature is not that urgend anymore. Priorities have moved it to the right. But I will come back, once this get in plan again and targetted ... Many thx for your support

[Frank Heimes (fheimes)]

Based on the last recent conversation (and comment #9) I am closing this ticket for now as Invalid.
If the upstream work succeeded and BPF got stabilized and tested again and further issues occur, a new LP bug can be opened any time.