Calico fails with "cni-relation-changed"

Bug #1997263 reported by Bas de Bruijne
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Calico Charm
New
Undecided
Unassigned

Bug Description

In testrun https://solutions.qa.canonical.com/v2/testruns/12fc360e-61e2-4f31-9776-f36d7d949bea, calico fails with:

```
kubernetes-worker/1 active idle 1 10.244.8.132 80/tcp,443/tcp Kubernetes worker running (without gpu support).
  calico/2 error idle 10.244.8.132 hook failed: "cni-relation-changed"
  containerd/2 active idle 10.244.8.132 Container runtime available
  nrpe/7 blocked idle 10.244.8.132 Nagios server not configured or related
kubernetes-worker/2 active idle 2 10.244.8.129 80/tcp,443/tcp Kubernetes worker running.
  calico/3 waiting idle 10.244.8.129 Waiting to retry disabling VXLAN TX checksumming
  containerd/3 active idle 10.244.8.129 Container runtime available
  nrpe/10 blocked idle 10.244.8.129 Nagios server not configured or related
```

In the calico logs we see:
```
unit-calico-2: 20:06:32 DEBUG unit.calico/2.juju-log cni:12: Writing file /lib/systemd/system/calico-node.service root:root 444
unit-calico-2: 20:06:32 WARNING unit.calico/2.cni-relation-changed Created symlink /etc/systemd/system/multi-user.target.wants/calico-node.service → /lib/systemd/system/calico-node.service.
unit-calico-2: 20:06:33 DEBUG unit.calico/2.juju-log cni:12: tracer: cleared flag cni.kubeconfig.changed
unit-calico-2: 20:06:33 DEBUG unit.calico/2.juju-log cni:12: tracer>
tracer: set flag calico.service.installed
tracer: ++ queue handler reactive/calico.py:172:check_etcd_changes
tracer: ++ queue handler reactive/calico.py:666:disable_vxlan_tx_checksumming
tracer: -- dequeue handler reactive/calico.py:248:install_calico_service
unit-calico-2: 20:06:33 INFO unit.calico/2.juju-log cni:12: Invoking reactive handler: reactive/calico.py:484:configure_node
unit-calico-2: 20:06:33 INFO unit.calico/2.juju-log cni:12: status-set: maintenance: Configuring Calico node
unit-calico-2: 20:06:41 ERROR unit.calico/2.juju-log cni:12: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-calico-2/charm/reactive/calico.py", line 494, in configure_node
    node = calicoctl_get('node', node_name)
  File "/var/lib/juju/agents/unit-calico-2/charm/reactive/calico.py", line 685, in calicoctl_get
    result = yaml.safe_load(output)
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/yaml/__init__.py", line 162, in safe_load
    return load(stream, SafeLoader)
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/yaml/constructor.py", line 49, in get_single_data
    node = self.get_single_node()
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/yaml/composer.py", line 39, in get_single_node
    if not self.check_event(StreamEndEvent):
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/yaml/parser.py", line 98, in check_event
    self.current_event = self.state()
  File "/var/lib/juju/agents/unit-calico-2/.venv/lib/python3.8/site-packages/yaml/parser.py", line 171, in parse_document_start
    raise ParserError(None, None,
yaml.parser.ParserError: expected '<document start>', but found '<block mapping start>'
  in "<byte string>", line 2, column 1:
    apiVersion: projectcalico.org/v3
    ^
```

It looks like the calico get node output is different than expected, I'm not sure why this is happening.

Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/12fc360e-61e2-4f31-9776-f36d7d949bea/index.html

Revision history for this message
George Kraft (cynerva) wrote :

Looks like the same issue as https://bugs.launchpad.net/charm-calico/+bug/1990672. I've marked this as a duplicate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.