Auto Package Testing

fwupd: network-based tests failed to run

Bug #2021908 reported by Zixing Liu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Auto Package Testing
Expired
Undecided
Unassigned
fwupd (Ubuntu)
Won't Fix
Undecided
Unassigned

Bug Description

`fwupd` package added network-based tests (to test whether firmware updates could be performed) since version 1.9.1.

The tests failed due to the HTTP proxy blocklist (which needs access to cdn.fwupd.org).

The issue was discovered in the local runner with these log lines:

..snip
failed to download http://cdn.fwupd.org/downloads/01b95b0206f1a42a2bf95a432d162ef1f9f1f71edb5696127c923ceffadfdf68-a3bu-xplained123.zip: Failed to download, server response was 403: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta type="copyright" content="Copyright (C) 1996-2019 The Squid Software Foundation and contributors">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>ERROR: The requested URL could not be retrieved</title>

In the Ubuntu-hosted autopkgtest runners, it failed with a cryptic message:

...snip
failed to load emulation data: emulation is not allowed from config
FAIL: fwupd/fwupd.test (Child process exited with code 1)

Tags: mantic adt-297
Revision history for this message
Brian Murray (brian-murray) wrote :

We should definitely look into the logging and why it is more cryptic in the autopkgtest environment.

However, I also wonder if the test would succeed in lgw01. This is something that could be tested from the staging environment.

Revision history for this message
Mario Limonciello (superm1) wrote :

I noticed that a bunch of the test runs blocking promotion passes, but some failed.

Including some jobs that failed once and hit retry and they pass.

So perhaps this is specific to certain autopkgtest hosts?

Revision history for this message
Brian Murray (brian-murray) wrote :

@Mario - Could you elaborate on the issues you are seeing with the autopkgtest environment including links to log files and examples? My team and I would like to resolve any issues you are seeing.

Revision history for this message
Mario Limonciello (superm1) wrote :

Look at proposed migration.

Here's two:

https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/amd64/f/fwupd/20230518_091353_f87ab@/log.gz

https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/arm64/f/fwupd/20230514_021241_b9c5e@/log.gz

Revision history for this message
Steve Langasek (vorlon) wrote :

Mario, you asserted that there were jobs that "failed once and hit retry and they pass". But the links you posted are to the test runs for fwupd 1.9.1-1, which have never passed. Are you saying that there were individual tests within the two runs of 1.9.1-1/amd64 that demonstrated flakiness? (if so, which ones?) or are you referring to something else, such as the tests passing on arm64 but not amd64?

Revision history for this message
Mario Limonciello (superm1) wrote :

Yes, but the logs are destroyed when you hit retry so I don't have the logs of fail followed by pass.

Revision history for this message
Steve Langasek (vorlon) wrote : Re: [Bug 2021908] Re: fwupd: network-based tests failed to run

On Tue, Jun 06, 2023 at 10:37:02PM -0000, Mario Limonciello wrote:
> Yes, but the logs are destroyed when you hit retry so I don't have the
> logs of fail followed by pass.

No, autopkgtest logs are not destroyed. That is only true for package build
logs.

Revision history for this message
Brian Murray (brian-murray) wrote : Re: [Canonical-ubuntu-qa] [Bug 2021908] Re: fwupd: network-based tests failed to run

On Tue, Jun 06, 2023 at 10:37:02PM -0000, Mario Limonciello wrote:
> Yes, but the logs are destroyed when you hit retry so I don't have the
> logs of fail followed by pass.

This is true of build logs but not autopkgtest log files. Here you can
see multiple failures with fwupd 1.9.1-1 on amd64.

https://autopkgtest.ubuntu.com/packages/fwupd/mantic/amd64

--
Brian Murray

Revision history for this message
Mario Limonciello (superm1) wrote :

Thanks for sharing link. Here's one that failed and I retried and it worked.

https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/ppc64el/f/fwupd/20230513_202849_b9c5e@/log.gz

https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/ppc64el/f/fwupd/20230514_032026_71fd9@/log.gz

Revision history for this message
Brian Murray (brian-murray) wrote (last edit ):

Thanks for the links Mario, this is quite helpful.

In one of the test runs, https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/ppc64el/f/fwupd/20230513_202849_b9c5e@/log.gz, we can see fwupd.test fail downloading.

Downloading… [*************** ]Downloading… [*************** ]Downloading…: 100%

failed to load emulation data: emulation is not allowed from config
FAIL: fwupd/fwupd.test (Child process exited with code 1)

From the begining of the log we can see that particular test was run in bos01.

autopkgtest [20:17:37]: host lrg-root4; command line: /home/ubuntu/autopkgtest/runner/autopkgtest --output-dir /tmp/autopkgtest-work.hbte3epz/out --timeout-copy=6000 --setup-commands /home/ubuntu/autopkgtest-cloud/worker-config-production/setup-canonical.sh --setup-commands /home/ubuntu/autopkgtest/setup-commands/setup-testbed --apt-pocket=proposed=src:fwupd --apt-upgrade fwupd --timeout-short=300 --timeout-copy=20000 --timeout-build=20000 --env=ADT_TEST_TRIGGERS=fwupd/1.9.1-1 -- ssh -s /home/ubuntu/autopkgtest/ssh-setup/nova -- --flavor autopkgtest --security-groups <email address hidden> --name adt-mantic-ppc64el-fwupd-20230513-201737-lrg-root4 --image adt/ubuntu-mantic-ppc64el-server --keyname testbed-lrg-root4 --net-id=net_prod-proposed-migration -e TERM=linux -e ''"'"'http_proxy=http://squid.internal:3128'"'"'' -e ''"'"'https_proxy=http://squid.internal:3128'"'"'' -e ''"'"'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,ports.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,launchpad.net,10.24.0.0/24'"'"'' --mirror=http://us.ports.ubuntu.com/ubuntu-ports/

However, the test run that passed, https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/ppc64el/f/fwupd/20230514_032026_71fd9@/log.gz, was run in bos02. So this is clearly is an issue with the proxy server configuration. I'll work on getting it sorted out.

Revision history for this message
Brian Murray (brian-murray) wrote :

I'll retry the s390x test as that also ran in bos01 as we might get lucky.

Getting the amd64 tests to pass will require some proxy server changes though.

Revision history for this message
Mario Limonciello (superm1) wrote :

As a fundamental question - do you think that pulling artifacts from LVFS by autopkgtest is reasonable? These were originally intended for gating upstream CI so that no code got introduced that broke existing devices that supported emulation data.

It's a LOT less likely that such breakage happens in distro packages that just move dependencies around.

TBH - It was a bit of a surprise that they were running in Ubuntu autopkgtest.

Revision history for this message
Paride Legovini (paride) wrote :

On the proxy settings: looks like this is one case where we set the proxy variables in autopkgtest environment, but there's is a daemon involved (fwupd) with its own environment where proxy vars are not set (same situation we have e.g. with Docker).

I had a quick look at fwupd and apparently it uses GProxyResolver, which is a GLib thing. I'm not sure on how to manually configure an http/https proxy so that GProxyResolver will use it.

Revision history for this message
Mario Limonciello (superm1) wrote :

It's not the daemon that downloads stuff, it's the client (via fwupdmgr). This should respect the environment variables http_proxy and https_proxy, so perhaps they're not set in the process environment for some reason.

Can you set them in /etc/environment perhaps?

Revision history for this message
Paride Legovini (paride) wrote :

I am not sure anymore it's a proxy issue; can we make a step back and re-check that conclusion?

From the test logs I don't see obvious "failed to download" errors, on the contrary looks like the download is working. Moreover looks like to me that fwupd/1.9.1-1 always failed on s390x regardless of the datacenter (bos01, bos02).

The bug description says that a "local run" hit the proxy issue, and concludes that the failures on the autopkgtest infrastructure fail for the same reason, but with a more cryptic message. To me in looks like a different failure.

Revision history for this message
Brian Murray (brian-murray) wrote :

I also think it is no longer, but was at one point, a proxy issue. The amd64 tests have now passed although we did not make any changes to the proxy configuration.

Additionally, as Paride mentioned the tests are failing on s390x regardless of the data center. Does s390x have the necessary capabilities to be able to run the fwupd.test?

In https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/s390x/f/fwupd/20230619_091825_be245@/log.gz (and another failure log) I see the following:

158s Wacom Intuos BT-M: failed after 10 retries: failed to SetReport: no matching event for ControlTransfer:Direction=0x01,RequestType=0x01,Recipient=0x01,Request=0x09,Value=0x03db,Idx=0x0000,Data=2wgAyYINBw==,Length=0x7

Revision history for this message
Mario Limonciello (superm1) wrote :

This looks like a real failure now with the emulated "Wacom Intuos" device. I'll bring it upstream.

Revision history for this message
Mario Limonciello (superm1) wrote :

Any chance you can test the potential fix on the real hardware without needing to make a round trip to the archive to see if it works?

https://github.com/fwupd/fwupd/pull/5929

Revision history for this message
Steve Langasek (vorlon) wrote :

working on building it in a ppa here https://launchpad.net/~canonical-foundations/+archive/ubuntu/ppa (just have to wait a cycle because I set up the new ppa with the wrong options initially) then will trigger an autopkgtest for it.

Revision history for this message
Steve Langasek (vorlon) wrote :

https://autopkgtest.ubuntu.com/results/autopkgtest-mantic-vorlon-ppa/mantic/s390x/f/fwupd/20230623_011450_1ce26@/log.gz shows a failure:

[...]
165s failed to load emulation data: emulation is not allowed from config
165s FAIL: fwupd/fwupd.test (Child process exited with code 1)
[...]
165s Update the device hash database...
165s failed to find 08d460be0f1f9f128413f816022a6439e0078018
165s Jun 23 00:58:44 autopkgtest systemd[1]: Starting fwupd.service - Firmware update daemon...
[...]
165s FAIL: fwupd/fwupdmgr.test (Child process exited with code 3)
[...]
166s SUMMARY: total=3; passed=1; skipped=0; failed=2; user=0.3s; system=0.1s; maxrss=26496
166s FAIL: fwupd/fwupd.test (Child process exited with code 1)
166s FAIL: fwupd/fwupdmgr.test (Child process exited with code 3)
166s autopkgtest [00:58:49]: test ci: -----------------------]
167s ci FAIL non-zero exit status 2

That's apparently more failures than in the previous version of the package?

Revision history for this message
Mario Limonciello (superm1) wrote :

To me; looks like the same proxy issue as before.

Revision history for this message
Zixing Liu (liushuyu-011) wrote :

The `failed to find 08d460be0f1f9f128413f816022a6439e0078018` part is suspicious.
The `failed to load emulation data` was likely the old network/proxy problem.

The patch only included the endianness handling for a particular device. The hash is probably in the wrong endianness order, I suppose?

Revision history for this message
Zixing Liu (liushuyu-011) wrote :

On line 749 of the https://autopkgtest.ubuntu.com/results/autopkgtest-mantic-vorlon-ppa/mantic/s390x/f/fwupd/20230623_011450_1ce26@/log.gz log, it showed:

```
452s FuEngine plugins disabled: test, test_ble
```

Somehow the test plugins were disabled. This likely caused the failures.

Zixing Liu (liushuyu-011)
tags: added: update-excuse
tags: added: mantic
Revision history for this message
Mario Limonciello (superm1) wrote :

Because of this in fwupd 1.9.3 release we've disabled network tests by default, and CI test suites will need to opt in.

https://github.com/fwupd/fwupd/commit/1d1e03fe02ad4b4bea41e9f7888d37fceb351174

1.9.3 is now in Debian unstable and mantic/proposed and I confirmed it's working.

If the autopkgtest infrastructure proxy issue is fixed we can re-enable it in Ubuntu by using CI_NETWORK=1 in the environment.

Changed in fwupd (Ubuntu):
status: New → Won't Fix
Brian Murray (brian-murray)
tags: added: adt-297
Revision history for this message
Brian Murray (brian-murray) wrote :

I've tested using wget with the proxy server configured to download a file from fwupd.org in both data centers and did not encounter an issue.

http_proxy=http://squid.internal:3128 https_proxy=http://squid.internal:3128 wget http://cdn.fwupd.org/downloads/01b95b0206f1a42a2bf95a432d162ef1f9f1f71edb5696127c923ceffadfdf68-a3bu-xplained123.zip

Could you provide me some command line arguments to pass fwupdmgr that would test the connectivity to fwupd.org?

Thanks!

Revision history for this message
Mario Limonciello (superm1) wrote :

fwupdmgr refresh would download content from the CDN.

Revision history for this message
Brian Murray (brian-murray) wrote :

Using fwupdmgr refresh also worked in both data centers, here is one example:

ubuntu@creation-test-1:~$ fwupdmgr refresh
Updating lvfs
Downloading… [ | ]
failed to download file: Failed to connect to cdn.fwupd.org port 443 after 60000 ms: Timeout was reached
ubuntu@creation-test-1:~$ http_proxy=http://squid.internal:3128 https_proxy=http://squid.internal:3128 fwupdmgr refresh
Updating lvfs
Downloading… [***************************************]
Successfully downloaded new metadata: 0 local devices supported

Revision history for this message
Paride Legovini (paride) wrote :

To reiterate on comment 15, I believe the failure is not due to a proxy issue.

Changed in auto-package-testing:
status: New → Incomplete
Changed in fwupd (Ubuntu):
status: Won't Fix → New
Brian Murray (brian-murray)
tags: removed: update-excuse
Revision history for this message
Mario Limonciello (superm1) wrote :

We're not going to re-introduce those tests in fwupd outside of upstream CI.

Changed in fwupd (Ubuntu):
status: New → Won't Fix
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for Auto Package Testing because there has been no activity for 60 days.]

Changed in auto-package-testing:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.