bos03 does not have adt/ arm64 xenial images

[Paride Legovini]

One first strange thing: apparently for Xenial we don't have an auto-sync/ image, like we have for the other releases, e.g:

  auto-sync/ubuntu-bionic-18.04-arm64-server-20230607-disk1.img
  auto-sync/ubuntu-focal-20.04-arm64-server-20240513-disk1.img

however we have an image with no prefix:

  ubuntu-xenial-16.04-arm64-server-20211001-disk1.img

and build-adt-image appears to be able to find it. However the image fails to boot, and therefore build-adt-image fails to generate an adt/... Xenial image, and we're unable to process xenial arm64 tests on bos03.

A reproducer is on these lines (with the right $secgroup):

openstack server create --flavor autopkgtest --security-group $secgroup --image ubuntu-xenial-16.04-arm64-server-20211001-disk1.img --key-name testbed-juju-7f2275-prod-proposed-migration-environment-3 --network=net_prod-proposed-migration paride-delme-foo

The instance is created, but is not reachable via ssh (this fails: nc -zv <ip> 22). It is not a networking issues as the same with --image auto-sync/ubuntu-bionic-18.04-arm64-server-20230607-disk1.img works.

We should probably get IS to add xenial to the list of images to auto-sync. Hopefully we'll get a working image, and then a working adt/ image will be generated.

[Andreas Hasenack (ahasenack)]

Example of error: https://autopkgtest.ubuntu.com/results/autopkgtest-xenial/xenial/arm64/u/ubuntu-advantage-tools/20240524_122900_98208@/log.gz

  0s autopkgtest [12:28:58]: starting date and time: 2024-05-24 12:28:58+0000
  0s autopkgtest [12:28:58]: git checkout: 699e7f9f ssh-setup/nova: explicitely set 'fqdn' in cloud-init
  0s autopkgtest [12:28:58]: host juju-7f2275-prod-proposed-migration-environment-3; command line: /home/ubuntu/autopkgtest/runner/autopkgtest --output-dir /tmp/autopkgtest-work.y8baphjq/out --timeout-copy=6000 --setup-commands /home/ubuntu/autopkgtest-cloud/worker-config-production/setup-canonical.sh --apt-pocket=proposed=src:ubuntu-advantage-tools --apt-upgrade ubuntu-advantage-tools --timeout-short=300 --timeout-copy=20000 --timeout-build=20000 '--env=ADT_TEST_TRIGGERS=ubuntu-advantage-tools/32.2~16.04' -- ssh -s /home/ubuntu/autopkgtest/ssh-setup/nova -- --flavor autopkgtest --security-groups <email address hidden> --name adt-xenial-arm64-ubuntu-advantage-tools-20240524-121844-juju-7f2275-prod-proposed-migration-environment-3-4f13bd91-fd36-4638-a6ad-77391b85e1a3 --image adt/ubuntu-xenial-arm64-server --keyname testbed-juju-7f2275-prod-proposed-migration-environment-3 --net-id=net_prod-proposed-migration -e TERM=linux -e ''"'"'http_proxy=http://squid.internal:3128'"'"'' -e ''"'"'https_proxy=http://squid.internal:3128'"'"'' -e ''"'"'no_proxy=127.0.0.1,127.0.1.1,login.ubuntu.com,localhost,localdomain,novalocal,internal,archive.ubuntu.com,ports.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,keyserver.ubuntu.com,launchpadlibrarian.net,launchpadcontent.net,launchpad.net,10.24.0.0/24,keystone.ps5.canonical.com,objectstorage.prodstack5.canonical.com'"'"'' --mirror=http://ftpmaster.internal/ubuntu/
  1s No nova image available that matches adt/ubuntu-xenial-arm64-server
  1s No UUID given. Instance won't be deleted!
  1s <VirtSubproc>: failure: setup script failed with code 1: /home/ubuntu/autopkgtest/ssh-setup/nova open --flavor autopkgtest --security-groups <email address hidden> --name adt-xenial-arm64-ubuntu-advantage-tools-20240524-121844-juju-7f2275-prod-proposed-migration-environment-3-4f13bd91-fd36-4638-a6ad-77391b85e1a3 --image adt/ubuntu-xenial-arm64-server --keyname testbed-juju-7f2275-prod-proposed-migration-environment-3 --net-id=net_prod-proposed-migration -e TERM=linux -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,login.ubuntu.com,localhost,localdomain,novalocal,internal,archive.ubuntu.com,ports.ubuntu.com,security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,keyserver.ubuntu.com,launchpadlibrarian.net,launchpadcontent.net,launchpad.net,10.24.0.0/24,keystone.ps5.canonical.com,objectstorage.prodstack5.canonical.com' --mirror=http://ftpmaster.internal/ubuntu/
  1s autopkgtest [12:28:59]: ERROR: testbed failure: unexpected eof from the testbed

Previous runs for 31.2.3~16.04 and even 32~16.04 more recently worked: https://autopkgtest.ubuntu.com/packages/ubuntu-advantage-tools/xenial/arm64

[Paride Legovini (paride)]

The runs that recently worked are not on bos03, for example

https://autopkgtest.ubuntu.com/results/autopkgtest-xenial/xenial/arm64/u/ubuntu-advantage-tools/20240416_195935_85bc6@/log.gz

is on bos01. This issue is specific to bos03.

[Paride Legovini (paride)]

@Andreas I've been reminded that arm64 is not an ESM supported architecture for Xenial, so we should not care about test results for this specific case. Britney may still flag the test as a failure, but the SRU team should know to ignore it.

Ideally we should not be attempting a test run, so this bug is in part still valid, but it is not as bad as I initially thought.

Do we agree on this?

[Andreas Hasenack (ahasenack)]

I think Product Management should be involved then, because we are releasing Pro for arm64 on xenial, and have been for a while. We therefore should test it on arm64 (and the other architectures we release it on).

And, as we saw throughout this bug, there have been good runs on xenial/arm64 in the past.

I'm starting an internal discussion.

[Steve Langasek (vorlon)]

On Fri, May 24, 2024 at 04:17:39PM -0000, Andreas Hasenack wrote:
> I think Product Management should be involved then, because we are
> releasing Pro for arm64 on xenial, and have been for a while. We
> therefore should test it on arm64 (and the other architectures we
> release it on).

I'm not sure what you mean by "releasing pro", but
https://ubuntu.com/security/esm explicitly says that only amd64 and s390x
have ESM support on Xenial.

[Steve Langasek (vorlon)]

On Fri, May 24, 2024 at 04:17:39PM -0000, Andreas Hasenack wrote:
> I think Product Management should be involved then, because we are
> releasing Pro for arm64 on xenial, and have been for a while. We
> therefore should test it on arm64 (and the other architectures we
> release it on).

Perhaps what you mean is that we publish the ubuntu-advantage-tools package
on arm64 in xenial.

But it's an architecture: all package, so per-arch tests are of little
significance. And xenial/arm64 is out of security support. So no users
should be running xenial on arm64, and arm64 autopkgtests shouldn't be a
blocker.

[Andreas Hasenack (ahasenack)]

> But it's an architecture: all package,

That's the metapackage (bin:ubuntu-advantage-tools). The actual binary is ubuntu-pro-client and is arch any.

[Andreas Hasenack (ahasenack)]

> I'm not sure what you mean by "releasing pro", but
> https://ubuntu.com/security/esm explicitly says that only amd64 and s390x
> have ESM support on Xenial.

Pro is not released through ESM, it goes into the regular updates pocket.

What I don't know is if other Pro offerings are a) available on xenial; b) and available for arm64 and other arches, on xenial

[Steve Langasek (vorlon)]

> > But it's an architecture: all package,

> That's the metapackage (bin:ubuntu-advantage-tools). The actual binary
> is ubuntu-pro-client and is arch any.

Fair enough, apparently I'm failing to operate rmadison correctly.

On Fri, May 24, 2024 at 06:13:20PM -0000, Andreas Hasenack wrote:
> > I'm not sure what you mean by "releasing pro", but
> > https://ubuntu.com/security/esm explicitly says that only amd64 and s390x
> > have ESM support on Xenial.

> Pro is not released through ESM, it goes into the regular updates
> pocket.

> What I don't know is if other Pro offerings are a) available on xenial;
> b) and available for arm64 and other arches, on xenial

Security support is a prerequisite for any other Pro offerings. Given that
ESM is not available for xenial/arm64 and xenial is out of standard security
support, none of the other product offerings can be available on arm64
either.

[Paride Legovini (paride)]

I filed LP: #2067406 to track the "bad UX" part of this bug, that is: when a test is requested for an unsupported configuration, autopkgtest-cloud should refuse to run it with a clear error message.

I believe we now agree the rest of this bug is Invalid; I'm marking it as such. If I'm mistaken on this, don't hesitate to move this back to New and reopen the discussion. Thanks!