slow downloading from registry.jujucharms.com

[Alex Lutay]

Hi,

This is a Juju bugreport based on https://warthogs.atlassian.net/browse/DPE-1660

At the moment K8s charms (OCI resources) downloading speed is slow from registry.jujucharms.com.
It causes bugreports to our charm as it stays in "unknown" state for 2-4 minutes:

> Normal Pulled 26s kubelet Successfully pulled image "registry.jujucharms.com/charm/kotcfrohea62xreenq1q75n1lyspke0qkurhk/postgresql-image@sha256:d48d9a008fb9426a898ff6adcc601978b2d4bbbf99b2e0a90ba8df8f98c3b9f7" in 4m10.451608962s (4m11.867654465s including waiting)

Please find detailed STR in https://warthogs.atlassian.net/browse/DPE-1660?focusedCommentId=238556

It is important to mention several issues here:

1) lack of any end-user visibility about "resources (OCI)" downloading progress is a root of the issue IMHO.
E.g. Juju shows downloading progress for VM charms, but not for K8s.

2) downloading speed for postgresql OCI resource (175MB) is 2-4 minutes from registry.jujucharms.com
(downloading the same OCI from ghcr.io and hub.docker.com is ~9 seconds)

3) juju confuses users with set status "unknown", see the unit status lifetime:

> postgresql-k8s/0 waiting allocating installing agent # OK
> postgresql-k8s/0* maintenance executing 10.1.221.139 installing charm software # OK
> postgresql-k8s/0* unknown executing 10.1.221.139

AFAIK, "unknown" is set during OCI downloading.

4) different juju units downloads the same resource/OCI in parallel (case `juju deploy postgresql-k8s -n 5`). Nice to optimize it too (https://github.com/juju/juju/pull/13215 for K8s OCI).

Thank you!

[Juan M. Tirado (tiradojm)]

Thanks Alex for reporting this situation. The items you expose here are too many for a single bug report.

From our understanding number 4) seems to be a legitimate bug we should explore.

At this moment we have no control of the registry.jujucharms.com repo so there's nothing we can do right now to improve the speed there.

About 1) and 3), those can be added to our wishlist.

[Alex Lutay (taurus)]

Dear Juan, Thank you for the fast reply!

I agree, several issues are mixed here and can be branched as sub-tickets.
I was trying to show the complete picture.

Most probably, we have found a workaround to hide "unknown" status of the charm:
https://warthogs.atlassian.net/browse/DPE-1660?focusedCommentId=238667

---

The another idea in this huge bucket:

5) Nice to have ops-framework way to report back container/resource downloading progress from K8s.
We will use container.can_connect() for now, but we do not know the K8s status: downloading or staring, or ...
To create nicer workaround, it would be superb to show "Downloading progress" => container.resource_status(resource).
Sure separate ticket can be branched to GitHub if you wish. :-)

P.S. Let's keep this ticket for the issue described as "4)". Tnx!

Re: 1) I have requested IS team limits to share in https://rt.admin.canonical.com/Ticket/Display.html?id=157400

[Tom Haddon (mthaddon)]

Hi Alex, registry.jujucharms.com has been redeployed to a different cloud (about a month ago I think). In my own testing image downloads appear to be significantly faster. A very large image (approx 700M) which previously took over 9 minutes to download now took slightly over 4 minutes (this is to my local machine, likely much quicker to infrastructure with better bandwidth).

Can you confirm if you're also seeing improvements in download speed time?

[Alex Lutay (taurus)]

I have confirmed it some time ago: https://warthogs.atlassian.net/browse/DPE-1660?focusedCommentId=323611

The current downloading speed for me is 14 seconds VS 4 minutes in the past. Tnx!