Flaky test brotli/basic on focal amd64?

Bug #1945713 reported by Bryce Harrington
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
libsoup2.4 (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Won't Fix
Low
Unassigned

Bug Description

The brotli/basic test case periodically fails. It appears that a few retriggers can result in a pass:

    https://autopkgtest.ubuntu.com/packages/libs/libsoup2.4/focal/amd64

The failures look like this in autopkgtest logs:

    ERROR:../tests/brotli-decompressor-test.c:59:test_brotli: assertion failed ((char*)out_bytes->data == contents): ("***\nU" == "***\n")

I've replaced most of the text with '***' to highlight the difference is a 'U' character appended.

The test case is essentially doing a decompression of 'brotli-data/compressed.br' and comparing that it matches 'brotli-data/uncompressed.txt'. I've done this manually using the brotli cli, and the files do indeed have the same content. I've also built the package and run the test case locally in a focal lxc container on amd64:

    $ cd libsoup2.4-gu/debian/libsoup2.4-tests/usr/libexec/installed-tests/libsoup-2.4
    $ while : ; do ./brotli-decompressor-test | grep 'brotli/basic'; sleep 0.1; done

I left that to run several hundred cycles but no failures reported at all.

There are a bunch of instances of this failure, here's a few for reference:

    https://autopkgtest.ubuntu.com/results/autopkgtest-focal/focal/amd64/libs/libsoup2.4/20210602_152107_d1d56@/log.gz
    https://autopkgtest.ubuntu.com/results/autopkgtest-focal/focal/amd64/libs/libsoup2.4/20210531_201241_8b1be@/log.gz
    https://autopkgtest.ubuntu.com/results/autopkgtest-focal/focal/amd64/libs/libsoup2.4/20200508_211619_2aec8@/log.gz

That last link is the earliest instance I've been able to find.

You'll notice in the second two links they also have a failure "not ok 2 /ssl/tls-interaction", but the first link shows only the brotli failure.

Interestingly, I've not found instances of this failure against other architectures than amd64, nor have I seen it on releases other than focal. No dice from googling for bug reports or other reports of test failures with libsoup2.4 + brotli.

The test's code is doing this, basically:

        SoupBrotliDecompressor *dec = soup_brotli_decompressor_new ();
        do {
                result = g_converter_convert (G_CONVERTER (dec), in_buf, length, out_buf, sizeof out_buf, 0,
                                              &bytes_read, &bytes_written, &error);
                g_byte_array_append (out_bytes, out_buf, bytes_written);
  in_buf += bytes_read;
                length -= bytes_read;

 } while (result == G_CONVERTER_CONVERTED);

soup_brotli_decompressor's code is here:

    https://gitlab.gnome.org/GNOME/libsoup/-/blob/master/libsoup/content-decoder/soup-brotli-decompressor.c

But I'm not spotting anything that would obviously cause a 'U' to be randomly appended sometimes. And why 'U'? UINT? Unknown? User error?

There are some interesting bug reports in GNOME about brotli in libsoup's bug tracker (e.g. #106, #146, #119, and #193), but none that match this.

Bryce Harrington (bryce)
tags: added: update-excuse
Revision history for this message
Bryce Harrington (bryce) wrote :

As can be seen here, the package doesn't pass its own tests even when triggered against only itself:

https://autopkgtest.ubuntu.com/results/autopkgtest-focal/focal/amd64/libs/libsoup2.4/20211004_230503_77208@/log.gz

Revision history for this message
Bryce Harrington (bryce) wrote :
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

I spent some time trying to get to the bottom of the issue, but unfortunately I was not able to make much progress here.

As expected, the very first thing to do here is to reproduce the bug locally. I tried many things:

- I built the package locally and ran autopkgtest against it. Passed.

- I built the package inside a Focal LXD container and then ran autopkgtest there. Passed.

- Still inside the Focal LXD container, I tried to invoke the failing test manually. Passed.

- I know that Ubuntu's autopkgtest infra uses a Bionic host to run the tests. I created a Bionic VM, and inside it I git cloned the development version of autopkgtest. Then, I invoked autopkgtest against the Focal source package for libsoup2.4, using a Focal LXD container as the testbed. Passed.

I ran out of ideas on what to try locally, and given that Bryce was able to add a hint for the failure and unblock apache2, I am giving up (at least for now). The way I see it, a person interested in continuing debugging this could:

- Add debugging statements to the libsoup2.4 code, upload the modified package to a PPA, and run autopkgtests using the Ubuntu infra. This should be able to provide more useful information regarding the array manipulation that's happening in the test.

- If the above doesn't work, one could try to add debugging statements to the libglib code that's responsible for array manipulations.

Lastly, it's important to mention that this may very well be something unrelated to libsoup2.4/glib. It can even be a hardware issue, who knows...

Revision history for this message
Brian Murray (brian-murray) wrote :

This has been consistently passing in Jammy, so I'm going to set this to Fix Released.

https://autopkgtest.ubuntu.com/packages/libs/libsoup2.4/jammy/amd64

Changed in libsoup2.4 (Ubuntu):
status: New → Fix Released
Revision history for this message
Bryce Harrington (bryce) wrote :

That's true that it works fine in jammy, but the breakage does still occur sometimes on focal. In case others run across this issue on focal, probably worth keeping this bug report open for reference.

Revision history for this message
Sebastien Bacher (seb128) wrote :

I'm going to set as wontfix so it stops showing up in rlsbugs reports, the issue isn't important enough at this point for us to justify doing the work figuring out what was the fix and SRUing it to focal

Changed in libsoup2.4 (Ubuntu Focal):
status: New → Won't Fix
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers