autopkgtest failures in mantic

[Bryce Harrington]

Autopkgtests are failing consistently and persistently for all architectures in mantic for this package.

The error logs show "504 Gateway Time-out" which initially sounded like a flaky test issue that would resolve on retriggering, but ample retriggers have proven that to be not the case.

python-mechanicalsoup tends to get at least a new debian update each Ubuntu cycle, but for mantic it has not, and is still at 0.10.0-6, the same version as shipped with lunar. However, a number of dependencies of this package have changed, when comparing tests run against this version in mantic with the last passing version:

https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/amd64/p/python-mechanicalsoup/20230614_170318_4b006@/log.gz
Setting up python3-iniconfig (1.1.1-2) ...
Setting up python3-py (1.11.0-1) ...
Setting up python3-all (3.11.2-1) ... ### 3.10.6-1 before
Setting up python3-packaging (23.0-1) ... ### 21.3-1.1 before
Setting up python3-requests-mock (1.9.3-2) ...
Setting up python3-pluggy (1.0.0+repack-1) ...
Setting up libxslt1.1:amd64 (1.1.35-1) ...
Setting up python3-soupsieve (2.4.1-2) ... ### 2.3.2-1 before
Setting up python3-pytest (7.2.1-2) ... ### 7.1.2-2ubuntu1 before
Setting up python3-bs4 (4.12.2-2) ... ### 4.11.1-1 before
Setting up python3-pytest-mock (3.8.2-3) ... ### 3.8.2-2 before
Setting up python3-lxml:amd64 (4.9.2-1build1) ... ### 4.9.1-1build1 before
Setting up python3-mechanicalsoup (0.10.0-6) ...

Given that the current logs show more test failures than the logs from when it initially failed, I suspect there might be more than one issue at play. But at least a no-change rebuild seems like the first thing to try.

[Bryce Harrington (bryce)]

After the no-change rebuild, running against -proposed, the test logs show 6 FAILED tests, versus 14 previously. So, as I expected the rebuild may have helped but there seem to be additional problems.

When I run the no-change rebuild against mantic-release instead, with no triggers except itself, I see only 2 FAILED tests. Running autopkgtest locally in LXC I see these same 2 failures. These seem to be the original "504 Gateway Time-out" and "502 Bad Gateway" errors that I thought might be flaky infrastructure but clearly are not since they repro on my own hw. Debian's CI system appears to show these same two failures in its logs as well.

So, it looks like in addition to whatever the no-change rebuild fixed, there is also this gateway error, plus potentially new issues arising from one or more package updates currently in -proposed.

[Bryce Harrington (bryce)]

Manual testing in LXC:

$ mkdir /tmp/autopkgtest
$ cp -r tests/ /tmp/autopkgtest/
$ cd /tmp/autopkgtest/
$ python3.11 -m pytest -v

======================================== short test summary info =========================================
FAILED tests/test_browser.py::test_post - assert (504 == 200)
FAILED tests/test_stateful_browser.py::test_submit_online - AssertionError: assert False
FAILED tests/test_stateful_browser.py::test_form_noaction - TypeError: Unexpected keyword arguments passed to pytest.raises: message
FAILED tests/test_stateful_browser.py::test_referer_follow_link - mechanicalsoup.utils.LinkNotFoundError
FAILED tests/test_stateful_browser.py::test_download_link - mechanicalsoup.utils.LinkNotFoundError
FAILED tests/test_stateful_browser.py::test_download_link_nofile - mechanicalsoup.utils.LinkNotFoundError
FAILED tests/test_stateful_browser.py::test_download_link_to_existing_file - mechanicalsoup.utils.LinkNotFoundError
================================ 7 failed, 57 passed in 186.49s (0:03:06) ================================

[Bryce Harrington (bryce)]

The first testcase seems to fail only intermittently, I ran it 10 times to get a failure only once:

$ python3.11 -m pytest -v tests/test_browser.py::test_post
========================================== test session starts ===========================================
platform linux -- Python 3.11.4, pytest-7.2.1, pluggy-1.0.0+repack -- /usr/bin/python3.11
cachedir: .pytest_cache
rootdir: /tmp/autopkgtest
plugins: mock-3.8.2, requests-mock-1.9.3
collected 1 item

tests/test_browser.py::test_post FAILED [100%]

================================================ FAILURES ================================================
_______________________________________________ test_post ________________________________________________

httpbin = <utils.HttpbinRemote object at 0x7fb72b3e13d0>

    def test_post(httpbin):
        browser = mechanicalsoup.Browser()
        data = {'color': 'blue', 'colorblind': 'True'}
        resp = browser.post(httpbin + "/post", data)
> assert(resp.status_code == 200 and resp.json()['form'] == data)
E assert (504 == 200)
E + where 504 = <Response [504]>.status_code

tests/test_browser.py:149: AssertionError
======================================== short test summary info =========================================
FAILED tests/test_browser.py::test_post - assert (504 == 200)
=========================================== 1 failed in 10.27s ===========================================

The httpbin fixture is defined in tests/conftest.py as returning an HttpbinRemote() instance, which is defined in tests/utils.py as "http://httpbin.org". That website indeed seems to be online yet slow, and thus may account for why these tests are timing out. Indeed, utils.py includes a comment suggesting this is an intentional override:

class HttpbinRemote:
    """Drop-in replacement for pytest-httpbin's httpbin fixture
    that uses the remote httpbin server instead of a local one."""
    def __init__(self):
        self.url = "http://httpbin.org"

    def __add__(self, x):
        return self.url + x

If I install python3-pytest-httpbin and then comment out the httpbin fixture definition in conftest, then test_post seems to work better (it executes faster and no longer fails intermittently afaict), however the full testsuite still fails, presumably because the test cases still are not getting the data they're expecting.

[Launchpad Janitor (janitor)]

This bug was fixed in the package python-mechanicalsoup - 0.10.0-6build1

---------------
python-mechanicalsoup (0.10.0-6build1) mantic; urgency=medium

  * No-change rebuild for python 3.11, soupsieve 2.4, &tc.
    (LP: #2024601)

 -- Bryce Harrington <email address hidden> Wed, 21 Jun 2023 09:55:34 -0700