ubuntu_bpf test_verifier failure on G/azure, F/oracle (boundary-crossing range and others)

[Krzysztof Kozlowski]

Groovy/linux-azure 5.8.0-1034.36, failures of ubuntu_bpf.test_verifier:

119. 06/08 10:40:38 DEBUG| utils:0153| [stdout] #64/u bounds check after truncation of non-boundary-crossing range OK
120. 06/08 10:40:38 DEBUG| utils:0153| [stdout] #64/p bounds check after truncation of non-boundary-crossing range OK
121. 06/08 10:40:38 DEBUG| utils:0153| [stdout] #65/u bounds check after truncation of boundary-crossing range (1) FAIL
...
133. 06/08 13:12:39 DEBUG| utils:0153| [stdout] #65/p bounds check after truncation of boundary-crossing range (1) OK
134. 06/08 13:12:39 DEBUG| utils:0153| [stdout] #66/u bounds check after truncation of boundary-crossing range (2) FAIL
...
1317. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #886/u PTR_TO_STACK mixed reg/k, 3 OK
1318. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #886/p PTR_TO_STACK mixed reg/k, 3 OK
1319. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #887/u PTR_TO_STACK reg FAIL
...
1414. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #933/p unpriv: pass pointer to helper function OK
1415. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #934/u unpriv: indirectly pass pointer on stack to helper function OK
1416. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #934/p unpriv: indirectly pass pointer on stack to helper function FAIL
1417. 06/08 10:40:41 DEBUG| utils:0153| [stdout] Unexpected success to load!
...
1509. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #987/u map access: value_ptr += known scalar, upper oob arith, test 1 OK
1510. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #987/p map access: value_ptr += known scalar, upper oob arith, test 1 OK
1511. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #988/u map access: value_ptr += known scalar, upper oob arith, test 2 OK
1512. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #988/p map access: value_ptr += known scalar, upper oob arith, test 2 OK
1513. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #989/u map access: value_ptr += known scalar, upper oob arith, test 3 FAIL
...
1522. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #991/p map access: value_ptr -= known scalar, lower oob arith, test 2 OK
1523. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #992/u map access: value_ptr -= known scalar, lower oob arith, test 3 FAIL
1524. 06/08 10:40:41 DEBUG| utils:0153| [stdout] Unexpected success to load!
...
1535. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #996/u map access: value_ptr += known scalar, 3 OK
1536. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #996/p map access: value_ptr += known scalar, 3 OK
1537. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #997/u map access: value_ptr += known scalar, 4 FAIL
1538. 06/08 10:40:41 DEBUG| utils:0153| [stdout] Unexpected success to load!
...
1567. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #1010/u map access: value_ptr -= known scalar OK
1568. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #1010/p map access: value_ptr -= known scalar OK
1569. 06/08 10:40:41 DEBUG| utils:0153| [stdout] #1011/u map access: value_ptr -= known scalar, 2 FAIL
1570. 06/08 10:40:41 DEBUG| utils:0153| [stdout] Unexpected success to load!
...
06/08 10:40:41 DEBUG| utils:0153| [stdout] Summary: 1610 PASSED, 0 SKIPPED, 8 FAILED

[Krzysztof Kozlowski (krzk)]

Found on groovy/oracle 5.8.0-1032.33

[Khaled El Mously (kmously)]

This appears to be affecting the 5.4 kernels as well - though for some reason the ubuntu_bpf test suite did not run on the master 5.4 kernel.

I do see it on Oracle 5.4 at least: http://10.246.72.9/5.4.0-1047.51-oracle/focal-linux-oracle-oracle-amd64-5.4.0-VM.Standard2.1-ubuntu_bpf/ubuntu_bpf/results/ubuntu_bpf.test_verifier/debug/ubuntu_bpf.test_verifier.DEBUG.html

though in that case it is 5 failures not 8.

[Krzysztof Kozlowski (krzk)]

Found on focal/azure 5.4.0-1050.52

[Thadeu Lima de Souza Cascardo (cascardo)]

From a conversation with Guilherme, I told him that this is likely due to verifier changes that are due, but that were not followed up by correspondent test_verifier changes.

Here is some excerpt from that conversation, where some commits are referred:

"
My bet are these two 9df9f9349abf7ab830aaa8d9eeaf5dcc3ba97f82 and a29fd91d1ec403d8fcc61ea9e724fcdfa1f20dce.

@gpiccoli though it looks like the test has both strings, one for unpriv, other for priv, so maybe the check is now also failing the same way for priv.

@gpiccoli d7a5091351756d0ae8e63134313c455624e36a13, at least the bounds.c seem relevant.
"

[Guilherme G. Piccoli (gpiccoli)]

Cascardo, thanks for the advice! We should check this to prevent such failures in next cycles.
For now, seems to be affecting the majority (if not ALL) 5.4 kernels.

Just observed it on B-5.4/aws, cycle sru-20210531 .

[Krzysztof Kozlowski (krzk)]

Found on groovy/azure 5.8.0-1037.39

[Guilherme G. Piccoli (gpiccoli)]

Observed in G/oracle, sru-20210531 .

[Guilherme G. Piccoli (gpiccoli)]

Observed in G/oracle, sru-20210621 .

[Guilherme G. Piccoli (gpiccoli)]

Observed in G/aws, sru-20210621 .

[Guilherme G. Piccoli (gpiccoli)]

Observed in B/aws-5.4, sru-20210621 .

[Guilherme G. Piccoli (gpiccoli)]

Observed this on B/oracle-5.4, cycle sru-20210621 .

[Krzysztof Kozlowski (krzk)]

Found on bionic/azure-5.4 5.4.0-1056.58~18.04.1 (and previous cycle 5.4.0-1054.56~18.04.1)

[Po-Hsu Lin (cypressyew)]

This issue does not exist in current supported kernels anymore, closing this and remove the corresponding hints.