[regression] build phase tries to update the apt cache even when not needed

[Olivier Tilloy]

(originally reported at https://forum.snapcraft.io/t/call-for-testing-snapcraft-4-0/16956/53)

The chromium snap fails to build with snapcraft 4.0.4 on the Launchpad builders. This is a regression, it built fine with snapcraft 3.11.

This is because Launchpad sets up a proxy to allow the build to access the internet for a limited amount of time, and chromium takes a very long time to build (typically 8+ hours on amd64, and much longer on other architectures).

chromium’s snapcraft.yaml¹ uses part ordering (the after keyword) to make sure that all the parts that need to fetch data/packages from the internet do so before actually building chromium. Not all the parts are specified in this ordering because not all of them actually stage packages or pull any source from the internet.

But it appears that even for those parts that don’t stage packages or pull anything, the new major version of snapcraft now runs apt-get update regardless, thus failing if the part is built after building chromium, because the proxy has been revoked. See the attached failed build log for details.

Here are the last few relevant lines:

Staging chromium
Pulling shared-mime-info
Pulling tests
Pulling xdg-email
Running build phase...
[12/Jun/2020:13:27:15 +0000] "GET http://ftpmaster.internal/ubuntu/dists/bionic/InRelease HTTP/1.1" 407 2077 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.1)"
Err:1 http://ftpmaster.internal/ubuntu bionic InRelease
  407 Proxy Authentication Required [IP: 10.10.10.1 8222]
[12/Jun/2020:13:27:15 +0000] "GET http://ftpmaster.internal/ubuntu/dists/bionic-security/InRelease HTTP/1.1" 407 2113 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.1)"
Err:2 http://ftpmaster.internal/ubuntu bionic-security InRelease
  407 Proxy Authentication Required [IP: 10.10.10.1 8222]
[12/Jun/2020:13:27:15 +0000] "GET http://ftpmaster.internal/ubuntu/dists/bionic-updates/InRelease HTTP/1.1" 407 2109 "-" "Debian APT-HTTP/1.3 (1.6.12ubuntu0.1)"
Err:3 http://ftpmaster.internal/ubuntu bionic-updates InRelease
  407 Proxy Authentication Required [IP: 10.10.10.1 8222]
Reading package lists...
EFailed to update the package cache: Some files could not be downloaded:

failed to run apt update

¹ https://git.launchpad.net/~chromium-team/chromium-browser/+git/snap-from-source/tree/snapcraft.yaml?h=stable

[Sergio Schvezov (sergiusens)]

Can you confirm that the build still works using Snapcraft from latest/stable/3.11 ?

[Olivier Tilloy (osomon)]

Yes, the build with snapcraft latest/stable/3.11 still works: https://code.launchpad.net/~chromium-team/+snap/chromium-snap-from-source-stable/+build/1000643.

I'm attaching the corresponding build log for future reference.

[Sergio Schvezov (sergiusens)]

I am trying to reproduce by doing a two step build (pull and snap) in a container and disconnecting the network entirely for the second phase and cannot seem to see any apt updates. I created https://github.com/snapcore/snapcraft/pull/3171 using latest edge which will give us debug output on launchpad.

[Sergio Schvezov (sergiusens)]

Looking at https://launchpadlibrarian.net/484588830/buildlog_snap_ubuntu_bionic_amd64_chromium-snap-from-source-stable_BUILDING.txt.gz it looks like install_build_packages is being called again and _check_if_all_packages_installed is not passing.

Traceback (most recent call last):
  File "/snap/snapcraft/4919/bin/snapcraft", line 30, in <module>
    sys.exit(load_entry_point('snapcraft==4.0.5+git5.g34b7491', 'console_scripts', 'snapcraft')())
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/click/core.py", line 1236, in invoke
    return Command.invoke(self, ctx)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/snapcraft/cli/_runner.py", line 98, in run
    snap_command.invoke(ctx)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/snapcraft/cli/_command.py", line 88, in invoke
    return super().invoke(ctx)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/snapcraft/cli/lifecycle.py", line 273, in snap
    _execute(steps.PRIME, parts=tuple(), pack_project=True, output=output, **kwargs)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/snapcraft/cli/lifecycle.py", line 78, in _execute
    lifecycle.execute(step, project_config, parts)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 113, in execute
    installed_packages = _install_build_packages(project_config.get_build_packages())
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/snapcraft/internal/lifecycle/_runner.py", line 58, in _install_build_packages
    return repo.Repo.install_build_packages(build_packages)
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/snapcraft/internal/repo/_deb.py", line 361, in install_build_packages
    cls.refresh_build_packages()
  File "/snap/snapcraft/4919/lib/python3.6/site-packages/snapcraft/internal/repo/_deb.py", line 303, in refresh_build_packages
    ) from call_error
snapcraft.internal.repo.errors.CacheUpdateFailedError: Failed to update the package cache: Some files could not be downloaded:

failed to run apt update

[Sergio Schvezov (sergiusens)]

I suspect this may be an issue in our code with virtual packages. Will look into details tomorrow.

[Olivier Tilloy (osomon)]

I'm attaching that failed build log with debug output for future reference.

[Sergio Schvezov (sergiusens)]

I can confirm that if I change libgcrypt-dev to libgcrypt20-dev that the cache invalidation works as expected.

[Sergio Schvezov (sergiusens)]

https://github.com/snapcore/snapcraft/pull/3177

[Olivier Tilloy (osomon)]

https://code.launchpad.net/~chromium-team/+snap/chromium-snap-from-source-stable/+build/1004957 is currently building against snapcraft from latest/edge/pr-3177.

[Sergio Schvezov (sergiusens)]

Can you try latest/edge/pr-3177 ?

[Olivier Tilloy (osomon)]

That build succeeded, thanks Sergio!