containerd does not respect juju model proxy

[Joshua Genet]

Related to this this bug:
https://bugs.launchpad.net/charm-kubernetes-worker/+bug/1841438

Using 1.16.2, during validation, test_audit_webhook tries and fails to pull a pod image due to not using the proxy.

"Error from server (BadRequest): container "test-audit-webhook" in pod "test-audit-webhook" is waiting to start: trying and failing to pull image."

The proxy is set in the juju model:
    juju-http-proxy: http://squid.internal:3128/
    juju-https-proxy: http://squid.internal:3128/
    apt-http-proxy: http://squid.internal:3128/
    apt-https-proxy: http://squid.internal:3128/
    snap-http-proxy: http://squid.internal:3128/
    snap-https-proxy: http://squid.internal:3128/
    juju-no-proxy: 10.244.0.0/15,192.168.0.0/16,172.16.0.0/12

Also setting the proxy in the containerd charm config resolves this issue:
    containerd:
        charm: cs:~containers/containerd
        options:
          http_proxy: http://squid.internal:3128/
          https_proxy: http://squid.internal:3128/
          no_proxy: 10.244.0.0/15,192.168.0.0/16,172.16.0.0/12

Here's the failing run:
https://solutions.qa.canonical.com/#/qa/testRun/71aeb8d7-49e1-41ea-8ea1-29b62ef3852e

And here's the successful run after setting the proxy in containerd:
https://solutions.qa.canonical.com/#/qa/testRun/6202c329-e927-40be-8727-e321ea9766ba

[George Kraft (cynerva)]

Again, this is almost certainly caused by large CIDRs used in the juju-no-proxy model config. See https://bugs.launchpad.net/charm-kubernetes-worker/+bug/1841438/comments/11

[Joshua Genet (genet022)]

This time, I don't think so. We pared down the 10.0.0.0/8 to 10.244.0.0/15. If you check out the bundles, the succeeding run has 10.244.0.0/15 in both the juju-no-proxy setting and passed into the containerd no_proxy. If the large CIDR was causing the failure, that would mean juju-no-proxy doesn't support it, but containerd does support large CIDRs. I assumed that's not the case, is it?

[George Kraft (cynerva)]

The /16, /15, and /12 CIDRs that you have in your juju-no-proxy are still large enough for the problem to occur.

CIDRs are "supported" when using juju-no-proxy.
CIDRs are not supported at all when using no_proxy.

Our attempt to support CIDRs in juju-no-proxy is what's breaking you here.

When you set both juju-no-proxy and no_proxy, the problem does not occur because no_proxy overrides juju-no-proxy, and we make no attempt to support CIDRs in no_proxy. So the proxy configuration no longer breaks when you set both, but you should know that containerd will completely ignore the CIDRs in no_proxy.

[Aymen Frikha (aym-frikha)]

containerd behind a proxy having trouble downloading the nvidia packages from nvidia.github.io even if we setup http_proxy and https_proxy configs. The only workaround I was able to find is adding manually the environment variable https_proxy in the debug-hook environment and run the install hook.

[Joseph Borg (joeborg)]

That was fixed in https://github.com/charmed-kubernetes/charm-containerd/pull/29

[Chris Sanders (chris.sanders)]

This is marked fixed-released for containerd but I'm still seeing this behavior on the stable channel rev 64 on a new deploy.

containerd:
  charm: cs:~containers/containerd
  options:
      gpu_driver: none

The proxy is set in the juju model:
    juju-http-proxy: http://squid.internal:3128/
    juju-https-proxy: http://squid.internal:3128/
    apt-http-proxy: http://squid.internal:3128/
    apt-https-proxy: http://squid.internal:3128/
    snap-http-proxy: http://squid.internal:3128/
    snap-https-proxy: http://squid.internal:3128/

However pulling containers fails until the config on the contarind is also set.

    containerd:
        charm: cs:~containers/containerd
        options:
          http_proxy: http://squid.internal:3128/
          https_proxy: http://squid.internal:3128/

The issue and solution is exactly the same as the original bug description. Is this not released to the charm store?

[Tim Van Steenburgh (tvansteenburgh)]

Hi Chris, yeah, there's something still broken here. It's not clear to me whether it was ever actually fixed or not. We're now tracking it in https://bugs.launchpad.net/charm-containerd/+bug/1874910.

[Tim Van Steenburgh (tvansteenburgh)]

It's also not clear whether the docker subordinate and/or docker registry charms are affected.

[Joseph Borg (joeborg)]

Tested on Docker subordinate. Working fine in 1.18 stable.

[Joseph Borg (joeborg)]

Also tested docker-registry (assuming that includes layer docker), working fine on 1.18 stable.