OCI arm64 build failure: Proxy Authentication Required

[Andreas Hasenack]

https://launchpad.net/~canonical-server/ubuntu-server-lma/+oci/grafana/+recipe/grafana-2/+build/119/+files/buildlog_oci_ubuntu_bionic_arm64_grafana-2_BUILDING.txt.gz

(...)
Removing intermediate container cfca72355cc4
 ---> 36650047f258
Step 15/37 : FROM golang:1.14.2 AS go-builder
[16/Aug/2020:16:46:02 +0000] "CONNECT registry-1.docker.io:443 HTTP/1.1" 407 1828 "-" "Go-http-client/1.1"
Get https://registry-1.docker.io/v2/: Proxy Authentication Required
Build failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_oci.py", line 136, in run
    self.build()
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_oci.py", line 126, in build
    self.run_build_command(args)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_oci.py", line 88, in run_build_command
    args, cwd=self.buildd_path, env=full_env, **kwargs)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/lxd.py", line 537, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['lxc', 'exec', 'lp-bionic-arm64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--', '/bin/sh', '-c', 'cd /home/buildd/grafana-2 && linux64 docker build --no-cache --build-arg http_proxy=http://10.10.10.1:8222/ --build-arg https_proxy=http://10.10.10.1:8222/ --tag grafana-2 --file Dockerfile.ubuntu /home/buildd/grafana-2']' returned non-zero exit status 1
Revoking proxy token...
Unable to revoke token for OCIRECIPEBUILD-119-1597588006: HTTP Error 401: UnauthorizedRUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=bionic --arch=arm64 OCIRECIPEBUILD-119
Scanning for processes to kill in build OCIRECIPEBUILD-119

I wonder if there is some proxy rule that gets temporarily added for the build, and is revoked before the build has a chance to finish? The arm64 build does take a lot of time:
- amd64: 1h14min
- arm64: failed in 2h23min with the above error
- ppc64el: 1h2min
- s390x: 47min

[Colin Watson (cjwatson)]

The timeout is two hours, so yes.

[Thiago F. Pappacena (pappacena)]

Just to keep this updated, there is an internal IS request (#127365) to increase the token TTL to 3h.

[Andreas Hasenack (ahasenack)]

Ticket link is https://portal.admin.canonical.com/C127365/, still not done

[Colin Watson (cjwatson)]

The timeout on production is three hours now.

[Andreas Hasenack (ahasenack)]

I got another failure, just over 3h:

https://launchpadlibrarian.net/502979893/buildlog_oci_ubuntu_bionic_arm64_grafana-ubuntu-layers_BUILDING.txt.gz