Calico charm doesn't support pulling images from insecure registries configured in containerd

Bug #1940299 reported by Nikolay Vinogradov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Calico Charm
New
Undecided
Unassigned

Bug Description

Hi.
The scenario described in this bug can be handy for testing in the development sandbox, as using insecure registries is not recommended in general.

However since containerd charm allows this, calico could potentially support this scenario as well.

Symptoms:
calico units are in error state. In the unit logs:

2021-08-17 13:36:07 INFO unit.calico/2.juju-log server.go:314 status-set: maintenance: Pulling calico-node image
2021-08-17 13:36:07 ERROR unit.calico/2.juju-log server.go:314 Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-calico-2/charm/reactive/calico.py", line 664, in pull_calico_node_image
    CTL.pull(image)
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/conctl/containerd.py", line 139, in pull
    return self._exec(*args)
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/conctl/containerd.py", line 29, in _exec
    return super()._exec(*['ctr'] + list(args))
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/conctl/base.py", line 30, in _exec
    return sub_run(args, stdout=PIPE, stderr=PIPE, check=True)
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('ctr', 'image', 'pull', '192.168.100.6:5000/cdk/calico/node:v3.10.1')' returned non-zero exit status 1.

Indeed:

$ sudo ctr image pull 192.168.100.6:5000/cdk/calico/node:v3.10.1
INFO[0000] trying next host error="failed to do request: Head https://192.168.100.6:5000/v2/cdk/calico/node/manifests/v3.10.1: http: server gave HTTP response to HTTPS client" host="192.168.100.6:5000"
ctr: failed to resolve reference "192.168.100.6:5000/cdk/calico/node:v3.10.1": failed to do request: Head https://192.168.100.6:5000/v2/cdk/calico/node/manifests/v3.10.1: http: server gave HTTP response to HTTPS client

However:

$ sudo ctr image pull --plain-http 192.168.100.6:5000/cdk/calico/node:v3.10.1
192.168.100.6:5000/cdk/calico/node:v3.10.1: resolved |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:a480c9291ece5a7b937780d740e5ab9343b4ed74653d449f3e294f0edb2fe8e2: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:71ec43cbc1b476f5f0bd6e6e414c0f07085953c8e100fbaace3100d5d5fca62b: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:4a88ba569c297526c461afba844ee8bb258274d54827db2e0eae232f60731903: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8d691f585fa8cec0eba196be460cfaffd69939782d6162986c3e0c5225d54f02: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:133ac0739597b2f608d638ea71c08fd93a17b9108394d6519df990966d339d30: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:210e2b599fb683e6d47228aa412545766dd912a67c9195cfb5e261f4d46f1894: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bbf3f5de750140ba341a5f85e1cd6a1fa8509625c597333fb73aad0f005e0763: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5123b94dd7dd89322708ef3b177e9d113af8bd5444a4318628a469f67d796f0b: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2128ac17051f541b22cd3b8f1e1b48ad65c3009e9c4f346d086427484e50a7d0: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bb042660b41da733d8e6cda2c427f3eeec1ed38565d8438f883c16f5686ba8b2: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:58f9b7ec65d341d8f029cbb3ea34123e0c6c021e918917712c019b2c1024d7e6: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:133cdc8b5d22fe741c9efccf0d97555b4950b4ecd05a3d6b67e34c8d8f38ee43: done |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d9253d8b0bfe10ff21945fc6743b540601a15e4e6a93bf12ef170fdc3734f042: done |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.1 s total: 0.0 B (0.0 B/s)
unpacking linux/amd64 sha256:a480c9291ece5a7b937780d740e5ab9343b4ed74653d449f3e294f0edb2fe8e2...
done: 5.875283ms

containerd config on this unit:
...
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
    [plugins.cri.registry.mirrors."192.168.100.6:5000"]
          endpoint = ["http://192.168.100.6:5000"]

      [plugins.cri.registry.auths]

        [plugins.cri.registry.auths."http://192.168.100.6:5000"]
          username = "admin"
          password = "password"
...
but 'ctr' client doesn't read this config, according to strace.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.