OCI: Failure to download a snap during build

[Sergio Durigan Junior]

We've been experiencing failures when building snap-based OCI images.

During the build process, one of the things we do in the Dockerfile is to download the snap (using curl). Here's the current curl incantation that we use (using the cassandra snap as an example):

curl -L $(curl -H 'X-Ubuntu-Series: 16' -H 'Snap-CDN: none' -H "X-Ubuntu-Architecture: $(dpkg --print-architecture)" 'https://api.snapcraft.io/api/v1/snaps/details/cassandra?channel=21.10/edge' | jq '.download_url' -r) --output cassandra.snap

With this, we're now seeing:

[0m[27/Oct/2021:19:07:54 +0000] "CONNECT canonical-lgw01.cdn.snapcraftcontent.com:443 HTTP/1.1" 403 1957 "-" "curl/7.74.0"
[91mcurl: (56) Received HTTP code 403 from proxy after CONNECT
[0m[27/Oct/2021:19:07:54 +0000] "CONNECT api.snapcraft.io:443 HTTP/1.1" 200 4653 "-" "curl/7.74.0"

Full logs here:

https://launchpadlibrarian.net/565900177/buildlog_oci_ubuntu_bionic_s390x_cassandra-21.10_BUILDING.txt.gz

I had a brief chat with Colin who suggested that we should be passing -H 'Snap-CDN: none' when downloading the snap. As seen above, even after providing this extra header the download still fails. It's also worth mentioning that this is the first time we're seeing such failures; all of our previous builds of snap-based OCIs worked fine.

[Sergio Durigan Junior (sergiodj)]

FWIW, I was passing the -H 'Snap-CDN: none' header to the wrong curl command; it's the outermost one that needs it, not the innermost. After fixing this nit, the build succeeded (at least for one of our snap-based images; I will rebuild the others soon, but I'm not expecting failures).

Thanks to Colin for the help.

[Colin Watson (cjwatson)]

OK, so I think this is worked around as far as this recipe is concerned, but I'd really like to see the store automatically redirect download requests from Launchpad builders to the non-CDN storage service; it would save a lot of faff trying to get `SNAPPY_STORE_NO_CDN=1` etc. into the right places (@wgrant's guidance was that we should avoid having builders go out to CDNs if we can). Adding a snapstore-server task for that.

[Colin Watson (cjwatson)]

See also https://forum.snapcraft.io/t/error-downloading-snap-stage-snaps-on-remote-build/26894.