Snap store APIs (api.snapcraft.io) are not reachable via IPv6

[Mattias Lindgren]

I am trying to refresh snaps in Ubuntu Core, but due to the fact that I don't have an IPv4 address on the system, I am unable to do so.

# snap refresh
error: cannot refresh []: cannot refresh snap-declaration for "nextcloud": Get
       https://assertions.ubuntu.com/v1/assertions/snap-declaration/16/njObIbGQEaVx1H4nyWxchk1i8opy4h54?max-format=2:
       dial tcp 162.213.33.203:443: connect: network is unreachable

Doing a host lookup confirms the issue

$ host assertions.ubuntu.com
assertions.ubuntu.com has address 162.213.33.203
assertions.ubuntu.com has address 162.213.33.202

I am using Ubuntu Core 16-2.26.14

What I expected to happen: snaps refresh
What happened intsead: assertions.ubuntu.com is unreachable over ipv6

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1710022/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Mark Shuttleworth (sabdfl)]

This makes me think we should be looking up 'assertions.snapcraft.io'
rather than 'assertions.ubuntu.com'

Mark

[Daniel Manrique (roadmr)]

How do you access IPv4-only resources from that system? Do you have some sort of gateway you could use to talk to the IPv4-only snap store services? I would suggest that as a workaround, since the work needed to make the snap store IPv6-ready will take some time, due to the fact there are many components in the stack and we need careful coordination of changes in all the pieces for enabling IPv6 support.

About the domain name: a good chunk of API endpoints are now on api.snapcraft.io (with a few older ones on dashboard.snapcraft.io), so it makes sense to allow access to the assertions services via that domain too. I'll make a note to look at what we need for that.

[Mattias Lindgren (mlindgren)]

I can do NAT64 in the meantime, but obviously not a long-term solution.

On Mon, Aug 14, 2017 at 2:11 PM, Daniel Manrique <
<email address hidden>> wrote:

> How do you access IPv4-only resources from that system? Do you have some
> sort of gateway you could use to talk to the IPv4-only snap store
> services? I would suggest that as a workaround, since the work needed to
> make the snap store IPv6-ready will take some time, due to the fact
> there are many components in the stack and we need careful coordination
> of changes in all the pieces for enabling IPv6 support.
>
> About the domain name: a good chunk of API endpoints are now on
> api.snapcraft.io (with a few older ones on dashboard.snapcraft.io), so
> it makes sense to allow access to the assertions services via that
> domain too. I'll make a note to look at what we need for that.
>
>
> ** Changed in: snapstore
> Status: New => Confirmed
>
> ** Changed in: snapstore
> Importance: Undecided => Medium
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1710022
>
> Title:
> assertions.ubuntu.com is not reachable via IPv6
>
> Status in Snap Store:
> Confirmed
>
> Bug description:
> I am trying to refresh snaps in Ubuntu Core, but due to the fact that
> I don't have an IPv4 address on the system, I am unable to do so.
>
> # snap refresh
> error: cannot refresh []: cannot refresh snap-declaration for
> "nextcloud": Get
> https://assertions.ubuntu.com/v1/assertions/snap-declaration/16/
> njObIbGQEaVx1H4nyWxchk1i8opy4h54?max-format=2:
> dial tcp 162.213.33.203:443: connect: network is unreachable
>
>
> Doing a host lookup confirms the issue
>
> $ host assertions.ubuntu.com
> assertions.ubuntu.com has address 162.213.33.203
> assertions.ubuntu.com has address 162.213.33.202
>
> I am using Ubuntu Core 16-2.26.14
>
> What I expected to happen: snaps refresh
> What happened intsead: assertions.ubuntu.com is unreachable over ipv6
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/snapstore/+bug/1710022/+subscriptions
>

[T Jeske (t-jeske)]

It's 2019 now and snapcraft is still not reachable via IPv6. I also have some IPv6-only machines (or some, where IPv6 is preferred as it doesn't count towards the traffic quota). I think it is about time this gets sorted. There is really not much to consider when turning on IPv6 on a webserver.

[Paweł Krawczyk (pawel-krawczyk)]

Agree with t-jeske above. Enabling IPv6 on a web server is not a rocket science really. This bug has been open for 2 years now.

[Daniel Manrique (roadmr)]

You make it sound like it's just a matter of assigning an IPv6 address to a host and tweaking a couple of Apache configs. It is not. The stack is more complex than that and changes are needed in other parts as well. We have this in our radar and will complete it as soon as feasible given other Snap Store priorities.

Cheers!

[Frank Meeusen (versionsix)]

Is there any information on when this will be put on the roadmap? Currently we are using a workaround using hardcoded nat64 endpoints to a tcp proxy It's 2020 now and native ipv6 support should be a no-brainer, especially for products targeted at IOT. I understand that until recently kubernetes could only drop ipv6 (by default), but the end customer should not be the victim of that poor architecture. There are other ways like putting a CDN in front.

[T Jeske (t-jeske)]

@Daniel
How is it more difficult than just assigning an IPv6 to the webserver? I don't need to know what you're doing internally. If IPv6 isn't a priority there, it's none of my concern. Also, it should be quite high on the priority list now, as IPv6-only machines are becoming more and more common and IPv4 addresses are getting more and more expensive. Right now, not offering IPv6 is starting to disable people in doing "stuff", contrary to enabling people what was once the benefit of using snap.

[James A. T. Rice (james-r-ubuntubugs)]

I'm also suffering from api.snapcraft.io only being reachable via legacy IP.

It seems there are other canonical hosts within that legacy /24 that do have IPv6, can the same treatment not be given to api.snapcraft.io?

$ host api.snapcraft.io
api.snapcraft.io has address 91.189.92.20
api.snapcraft.io has address 91.189.92.19
api.snapcraft.io has address 91.189.92.38
api.snapcraft.io has address 91.189.92.41

$ host kudan.canonical.com
kudan.canonical.com has address 91.189.92.150
kudan.canonical.com has IPv6 address 2001:67c:1360:8c01::1b

Thankyou

[Kevin Otte (nivex)]

Just had #1878679 marked as duplicate of this.

> How do you access IPv4-only resources from that system?

I don't. I am able to reach the Ubuntu package mirrors and all of my resources via IPv6.

> the end customer should not be the victim of that poor architecture

Amen! At SouthEast LinuxFest 2010 there was a presentation called "IPv6: No Longer Optional". In June 2012 we had the World IPv6 Launch. Snapcraft would not come along for another three years. IPv6 should have been baked in to the initial design.

Requiring every customer to maintain their own workaround to access the resources of a supposedly forward-looking technology company borders on insulting.

So while I appreciate that it is now "on the radar", I have little sympathy for the lack of forethought.

[The rest of this comment has been redacted as I've already done enough shouting into the void here.]

[grzchr15 (christian-bretterhofer)]

Add me as impacted

[Sly Gryphon (sgryphon)]

I ran across this trying to set up an IPv6 machine, and found that api.snapcraft.io does not support IPv6.

Unless it is doing something extremely funky, it should be as simple as adding an IPv6 address and setting up the AAAA record.

Currently there are a bunch of A records, 91.189.92.20, 91.189.92.41; there should be nothing in the stack that cares which IP address, or what type, the value resolves to. Just pass in the DNS name and get back whatever IP it gives you.

In fact, if DNS64+NAT64 works as a workaround then you can guarantee that it is as simple as sticking an IPv6 AAAA address on the server -- because that is exactly what you get with DNS64... an AAAA address, and as far as the client is concerned it is IPv6.

If you don't have dual stack to the actual machines, then sticking a reverse proxy in front of it that does have one is pretty simple (e.g. Caddy or Nginx), or there are CDNs that will enable it for you.

It is now 2020, 3 years after this bug was logged. 30% of the Internet now runs IPv6, which is a reasonably significant portion.

[Sly Gryphon (sgryphon)]

@Daniel Manrique (roadmr) wrote:

> " You make it sound like it's just a matter of assigning an IPv6 address to a host and tweaking a couple of Apache configs. "

If the work around you suggested in 2017, by using a gateway works, then it *is* as simple as sticking an IPv6 address on the domain name(s).

Because that is what a DNS64+NAT64 solution does -- the DNS64 just simply assigns an IPv6 address (a special, local address), and then the client talks to that IPv6 address.

If it wasn't that simple (e.g. something internal to the snap client that checked the IP address format), then your suggested workaround would not work.

It might be more complicated than one public DNS entry, if you use api.snapcraft.io, assertions.ubuntu.com, etc, but not much.

Note that anything internal to the stack is not relevant and can remain IPv4, e.g. if API calls something else via IPv4 it is irrelevant, all that matters is public IPv6 addresses for the client to call.

[Vivien GUEANT (vivienfr)]

Hello,

My backend servers only have IPv6 (I don't have DNS64/NAT64 to access IPv4 resources because I don't have any need except Snap Store). I can access APT repositories, but not Snap Store (api.snapcraft.io).

If you have problems putting the back-end server in IPv6, you can put in front of a proxy, which will allow to announce a public IPv6, which will then be internally translated to IPv4.

It is important that in 2021 the various resources of a server are available in IPv6, because IPv4 is absent on more and more backend servers.

Vivien

[Ihor Hordiichuk (hordiichuk)]

[James A. T. Rice (james-r-ubuntubugs) wrote on 2020-05-19: #10]
I'm also suffering from api.snapcraft.io only being reachable via legacy IP. It seems there are other canonical hosts within that legacy /24 that do have IPv6, can the same treatment not be given to api.snapcraft.io?

$ host api.snapcraft.io api.snapcraft.io has address 91.189.92.20
$ host kudan.canonical.com kudan.canonical.com has address 91.189.92.150
kudan.canonical.com has IPv6 address 2001:67c:1360:8c01::1b
******************************************************************

[Kevin Otte (nivex) wrote on 2020-06-03:]
> How do you access IPv4-only resources from that system?
I don't. I am able to reach the Ubuntu package mirrors and all of my resources via IPv6.
*******************************************************************
[**Vivien GUEANT (vivienfr) wrote on 2021-01-20: #15]
My backend servers only have IPv6 (I don't have DNS64/NAT64 to access IPv4 resources because I don't have any need except Snap Store). I can access APT repositories, but not Snap Store (api.snapcraft.io).
********************************************************************

I noticed interesting feature request on https://bugs.launchpad.net/snapstore-server/+bug/1916392/comments/3
"[Important proposal] Decentralized or Federated snap store operation structure similar ubuntu mirrors"

I believe we should urge canonical for start to work on "#1916392" instead this; actually it has more benefits, not only solve this problem. Snap CLI is similar to APT package manager and how it works so there's no reason for unable to have this!

This is workable with snap-store (gnome-software) GUI tool as also mentioned in the comment.

[Dolf Schimmel (Freeaqingme) (freeaqingme)]

I'm affected by this issue as well. I'm not entirely sure when people started developing snapd/snapcraft, but I reckon it was well beyond the introduction of IPv6. I hope that for new services people can take IPv6 into account from the get-go.

Maybe you can explain why this is so difficult to realize in the case of snapcraft.io? To someone not intimately familiar with its internals it doesn't seem all to difficult to add an extra reverse proxy?

[Graham Leggett (minfrin-y)]

We currently get the below on an IPv6 only host. Looks like snapstore.io is a legacy system, is it going away?

Connection to the Snap Store failed

You have the package lxd installed but your system is unable to reach
the Snap Store. lxd is now provided via a snap and the release
upgrade will fail if snapd is not functional. Please make sure you're
connected to the Internet and update any firewall or proxy settings
as needed so that you can reach api.snapcraft.io. If you are an
enterprise with a firewall setup you may want to configure a Snap
Store proxy.

Restoring original system state

Aborting

[j^ (j)]

So its not possible to use snap on IPv6 only offerings of various hosting providers. Interesting given that snap is needed to run lxd. Please fix this already.

[Kevin Otte (nivex)]

Unsubscribing from this bug. This was actually the last straw that caused my migration away from Ubuntu.

[Ted Spikes (ted-0)]

Issue still present.

What sucks is that this blocks me completely from installing certbot, had to rent a temporary ipv4 address from my hosting to get it working.

[Thiago Martins (martinx)]

No worries, folks!

I've heard Canonical will do this by around 2045 when the price of 1 IPv4 address reaches the price of 1 Bitcoin or the Singularity is achieved, then the super AI can implement IPv6 on Snap Store since humans don't know how to do it.

What a shame this is.

At least buying IPv4 sounds like an excellent investment! Until the IPv4 bubble pops...

[Daniele Baroncelli (daniele-baroncelli)]

Shame on Canonical!
Still happening now after 6 years!

[Federico Bosi (rhxto)]

I will stop using snap because of this. It's unbelievable how many packages are being migrated to snapcraft if they can't even keep up with 10+ year old technology.

[Earning Evolved (earningevolved)]

Kevin, this issue has nothing to do with Ubuntu! No matter what Linux version you are using, if your machine is ipv6 only, you can't use Snaps! Debian, Fedora, Arch, whatever - it doesn't matter. The issue is that snaps are still only working for ipv4 addresses and not working for ipv6 addresses.

Snap is so great on ipv4 systems. It is a shame it is 2023 and this is still an issue for ipv6 machines!

[Steffan Byrne (vermillion)]

Is there anywhere that we can track the work being done to support this?

At the end of July 2023, AWS announced that they planned to start charging for usage of ipv4 addresses after February 1st, 2024. To avoid this charge, I decided to migrate my nextcloud setup to a new instance running Jammy with ipv6-only networking. I use the wonderful nextcloud snap which is supposed to make it simple to backup and restore to a new host. I was incredibly disappointed when I discovered that I cannot reach the snap store from this new machine.

I anticipate many more users will run into similar scenarios this year as they try to migrate and avoid a charge for a networking detail that shouldn't be a roadblock. Please prioritize this work.

[Patrik Wenger (paddor)]

What's the status on this?

[Will Daniels (wdaniels)]

As predicted by Steffan Byrne above, I just came across this issue while changing all my AWS instances to use IPv6 only. I figured that I don't need to bother with IPv4 at all nowadays on Ubuntu servers that only need public IP transit to check and download updates... apparently that was optimistic!

Not impossible to work around, obviously, but an otherwise needless expense for a NAT gateway, and just frankly astounding in 2024 :D