snapd connectivity check did not change fastly cdn

[Dimitri John Ledkov]

on power during disco release sprint we have observed the following:

snapd able to access snapstore api, and searched / found updated subiquity.

But was unable to refresh to it, as there was no access to the fastly CDN.

It would be nice, if the snapd connectivity check checked both access to _both_ store(proxy)-api and the (no-)CDN download location too.

I haven't double check with most recent snapd.

What logs / information should I provide to confirm / deny this bug?

[Michael Hudson-Doyle (mwhudson)]

Ugh.

I don't see what subiquity can do differently here, we're just making snapd api calls, it seems to me (and I don't just want to be seen as passing the buck) that snapd is the right place to solve this (or even the store: it could detect the request is coming from an internal IP and not return CDN links?).

That said, being able to access api.snapcraft.com but not fastly is a situation mostly (?) peculiar to our infrastructure, so maybe we should add a way to specify via answers that we should set SNAPPY_STORE_NO_CDN=1 for snapd.

[Dimitri John Ledkov (xnox)]

I only marked it as affecting subiquity, in practice yes fixes need to happen in snapd.

I don't think that' uncommon to only our infrastructure. We have seen in the wild people enable the endspoints that are seen and only api.snapcraft.com is attempted (or so it would seem) without it being visible that fastly is needed too. It would be nice if snapd would try to hit both in parallel.

Or whatever the brandstore / store-proxy endpoints are.

[John Lenton (chipaca)]

On the snapd side, what we do does not "know" about fastly at all: what the connectivity check does is
1. hit the store's "info" endpoint asking about "core",
2. HEAD the download url provided in (1)

The request done in (2) sets the cdn headers as usual (ie it looks at whether CDN is disabled by the environment, and at the device auth context to see if it's overridden by cloud info).

You can see this logic spelled out
https://github.com/snapcore/snapd/blob/master/store/store.go#L2404

if you turn on SNAPD_DEBUG and SNPAD_DEBUG_HTTP=7 you should be able to see the requests as they go.

[John Lenton (chipaca)]

Importantly, the decision to go to e.g. fastly, vs nothing, is done _store side_. snapd knows nothing about fastly, and the store could (and has in the past) swapped CDNs and snapd is none the wiser.
Here's the log of a connectivity check, done from home:

https://pastebin.canonical.com/p/p7BNX2z85Q/

(that's the regular debug log output run through delog.py¹)
note that it only hits fastly because of a 302 from the store.

snapd does exactly what it does to fetch a snap by name. In that sense, snapd is already doing what this bug asks it to do. If the connectivity check passes, the download from snapd should pass just as well. If it doesn't, something else might be broken.

1. https://gist.github.com/chipaca/5d0f0e2b7fecd2df87f25b798a6c6537

[John Lenton (chipaca)]

Here's the same log on the ubuntu pastebin (not sure why i thought the bug was private): https://pastebin.ubuntu.com/p/dpxpPkHSbM/

[Michael Hudson-Doyle (mwhudson)]

Hm so it sounds like snapd's behaviour is what Dimitri is asking for here? Has this changed in snapd recently (note that this behaviour was observed during the disco release sprint, so April 2019).

[John Lenton (chipaca)]

No, snapd's behaviour has not changed

[Zygmunt Krynicki (zyga)]

I've added snapd project task and set this to triaged/wishlist but based on the current discussion I think WONTFIX or INVALID might also be appropriate.

[John Lenton (chipaca)]

Nah, on the contrary, there is nothing for snapd to do here; it already does what's wanted (and has from day one). I'm removing snapd from the bug ...

of course if I got something wrong, please add it back and let us know :-) (setting it to new will get it noticed soonest)

[Michael Hudson-Doyle (mwhudson)]

Is this still a current issue? I haven't heard anyone complain about it for a while now.

[Dimitri John Ledkov (xnox)]

I did have this very recently on our s390x box. I think I can reproduce this behaviour there!