Ubuntu CI that runs tests via autopkgtest for systemd on GitHub reports the wrong results

[Evgeny Vereshchagin]

I'm not sure this is the right place to report this, but apparently it's the best way to reach out to Dimitri John Ledkov (https://github.com/xnox) and Iain Lane (https://github.com/iainlane), who I believe at least know who maintains Ubuntu CI there.

I'm copying the following verbatim from https://github.com/systemd/systemd/pull/11531#issuecomment-464731263 (where we are kind of discussing the issue):

> The two PRs you referenced to fail in the `make check` unit tests (test-path).

That's correct that test-path failed but it failed in https://github.com/systemd/systemd/pull/11685 while Ubuntu CI reported it in https://github.com/systemd/systemd/pull/11686 (that is, in https://github.com/systemd/systemd/pull/11686 in the logs for some reason UPSTREAM_PULL_REQUEST is 11685 instead of 11686 and something like that has happened often lately).

In https://github.com/systemd/systemd/pull/11743#issuecomment-464637797, the issue is that Ubuntu CI showed the results for the first version of the PR (where there was a bug) and it didn't run the tests against the latest version. But in that case it's hard to tell because there is no way to figure out what exactly Ubuntu CI tested. That's why I asked @xnox to add `git describe` to https://salsa.debian.org/systemd-team/systemd/blob/master/debian/extra/checkout-upstream so that by looking at UPSTREAM_PULL_REQUEST and the output of `git describe` it would be possible to find out what Ubuntu CI reports.

[Martin Pitt (pitti)]

It seems to me that the logs are internally consistent, i. e. the mentioned UPSTREAM_PULL_REQUEST in the log does match the test results. But they get sent to the wrong PR, i. e. to the wrong statuses API.

E. g. https://api.github.com/repos/systemd/systemd/commits/99894b867f1293f56d181d62f5015c5a0a8adbda/status was triggered by https://github.com/systemd/systemd/pull/11682 but the referenced logs are for PR #11767. This caused the landing of a regression (that the tests would have noticed).

[Martin Pitt (pitti)]

Another example: https://github.com/systemd/systemd/pull/11802 refers to the correct amd64 log https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-upstream-systemd-ci-systemd-ci/bionic/amd64/s/systemd-upstream/20190222_161608_7fe1f@/log.gz .

But https://github.com/systemd/systemd/pull/11804 refers to the exact same log. It is also the only result for 11804: https://api.github.com/repos/systemd/systemd/statuses/b427959d65ba0edff385146d38825bb169458554

[Iain Lane (laney)]

Thanks, I would guess that I broke this somehow in https://git.launchpad.net/autopkgtest-cloud/commit/?id=58adff467c108506f979e229aa8e9e955a7d0d0a - which was a commit I made because systemd PRs were taking a long time after their jobs finished (several hours) to report any results due to that script having terrible runtime.

I'll stare at that code and see if I can see what I did wrong. Sorry about that.

[Iain Lane (laney)]

Yeah OK, I'm pretty sure that change is bogus. I think it'll mean that if we hit the 'cache' we'll post back to the wrong status URL, since we don't check the environment matches.

I'll just revert that for now. Sorry again.

[Iain Lane (laney)]

Done. Please let me know here if this is still a problem.

[Martin Pitt (pitti)]

Thanks Iain! I'll keep an eye on this.

[Dan Streetman (ddstreet)]

please reopen if this is still an issue