test_verifier in ubuntu_bpf failed with 4 test cases (#27, #30, #150, #797) on J-5.15 AMD64/s390x openstack

[Po-Hsu Lin]

Issue found with J-generic-5.15.0-88.98 and J-lowlatency-5.15.0-88.98 on Openstack AMD64 instance.

After the ubuntu_bpf build issue (bug 2035181) fixed, test in ubuntu_bpf can be tested and reporting failures:

 #27/u atomic compare-and-exchange smoketest - 32bit FAIL
 Unexpected success to load!
 verification time 36 usec
 stack depth 4
 processed 23 insns (limit 1000000) max_states_per_insn 0 total_states 2 peak_states 2 mark_read 1

 #30/u BPF_W cmpxchg should zero top 32 bits FAIL
 Unexpected success to load!
 verification time 17 usec
 stack depth 8
 processed 13 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 0

 #150/p calls: trigger reg2btf_ids[reg->type] for reg->type > __BPF_REG_TYPE_MAX FAIL
 Unexpected verifier log!
 EXP: arg#0 pointer type STRUCT prog_test_ref_kfunc must point
 RES:
 FAIL
 Unexpected error message!
     EXP: arg#0 pointer type STRUCT prog_test_ref_kfunc must point
     RES: kernel btf_id 0 is not a function
 verification time 10 usec
 stack depth 0
 processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

 kernel btf_id 0 is not a function
 verification time 10 usec
 stack depth 0
 processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

 #797/p precise: ST insn causing spi > allocated_stack FAIL
 Unexpected verifier log!
 EXP: parent didn't have regs=10 stack=0 marks
 RES:
 func#0 @0
 0: R1=ctx(id=0,off=0,imm=0) R10=fp0
 0: (bf) r3 = r10
 1: R1=ctx(id=0,off=0,imm=0) R3=fp0 R10=fp0
 1: (55) if r3 != 0x7b goto pc+0
  R1=ctx(id=0,off=0,imm=0) R3=fp0 R10=fp0
 2: R1=ctx(id=0,off=0,imm=0) R3=fp0 R10=fp0
 2: (7a) *(u64 *)(r3 -8) = 0
 3: R1=ctx(id=0,off=0,imm=0) R3=fp0 R10=fp0 fp-8_w=00000000
 3: (79) r4 = *(u64 *)(r10 -8)
 4: R1=ctx(id=0,off=0,imm=0) R3=fp0 R4_w=invP0 R10=fp0 fp-8_w=00000000
 4: (b7) r0 = -1
 5: R0=inv-1 R1=ctx(id=0,off=0,imm=0) R3=fp0 R4=invP0 R10=fp0 fp-8=00000000
 5: (2d) if r4 > r0 goto pc+0
 last_idx 5 first_idx 5
  R0_rw=invP-1 R1=ctx(id=0,off=0,imm=0) R3=fp0 R4_rw=invP0 R10=fp0 fp-8_w=00000000
 parent didn't have regs=1 stack=0 marks
 last_idx 4 first_idx 2
 regs=1 stack=0 before 4: (b7) r0 = -1
 6: R0=invP-1 R1=ctx(id=0,off=0,imm=0) R3=fp0 R4=invP0 R10=fp0 fp-8=00000000
 6: (95) exit

 from 1 to 2: safe
 processed 8 insns (limit 1000000) max_states_per_insn 0 total_states 4 peak_states 4 mark_read 1

This should not be considered as a regression since we just start testing ubuntu_bpf on openstack instances from 5.15.0-83 (and it's affected by the aforementioned build issue since then)

We used to have bug 1991351, but I decided to open a new one as that report is quite old.

[Po-Hsu Lin (cypressyew)]

These 4 failures can be found on J-5.15 openstack s390x VM as well.

[Po-Hsu Lin (cypressyew)]

This test used to fail with just 3 sub-test cases (#27, #30, #150) recorded in bug 1991351.

This is a new test failure:
 #797/p precise: ST insn causing spi > allocated_stack FAIL

Looking back into available test history, taking AWS c5n.large J-aws as baseline we can see that:
* 2023.01.30/jammy/linux-aws/5.15.0-1031.35 - failed with #27, #30, #150
* 2023.02.27/jammy/linux-aws/5.15.0-1033.37 - failed with #27, #30, #150
* 2023.03.20/jammy/linux-aws/5.15.0-1034.38 - failed with #27, #30, #150
* 2023.04.17/jammy/linux-aws/5.15.0-1036.40 - failed with #27, #30, #150
* 2023.05.15/jammy/linux-aws/5.15.0-1038.43 - failed with #27, #30, #150
* 2023.07.10/jammy/linux-aws/5.15.0-1042.47 - failed with #27, #30, #150
* 2023.08.07/jammy/linux-aws/5.15.0-1044.49 - failed with #27, #30, #150
* 2023.09.04/jammy/linux-aws/5.15.0-1046.51 - test blocked by a build issue (bug 2035181)
* 2023.09.04/jammy/linux-aws/5.15.0-1047.52 - failed with #27, #30, #150, #797
* 2023.10.02/jammy/linux-aws/5.15.0-1049.54 - failed with #27, #30, #150, #797

I think we should investigate this.

[Po-Hsu Lin (cypressyew)]

This issue mentioned in comment #2 has gone with 5.15.0-91 in sru-202301030
  #797/p precise: ST insn causing spi > allocated_stack FAIL