One of calico instance stuck in "Configuring Calico"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Calico Charm |
In Progress
|
High
|
Unassigned |
Bug Description
In this SQA run - https:/
When deploying Charmed Kubernetes on top of baremetal with charm Cailco rev 105 on focal,
one of the calico instance stuck in "Configuring Calico",
and kubernetes-
Error logs
2024-04-26 16:00:11 INFO unit.calico/
2024-04-26 16:00:11 ERROR unit.calico/
2024-04-26 16:00:11 ERROR unit.calico/
2024-04-26 16:00:11 ERROR unit.calico/
Traceback (most recent call last):
File "./src/charm.py", line 298, in _configure_node
node = self._calicoctl
File "./src/charm.py", line 640, in _calicoctl_get
output = self.calicoctl(
File "./src/charm.py", line 632, in calicoctl
return subprocess.
File "/usr/lib/
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/
raise CalledProcessEr
subprocess.
2024-04-26 16:00:11 ERROR unit.calico/
Traceback (most recent call last):
File "./src/charm.py", line 174, in _install_or_upgrade
self.
File "./src/charm.py", line 125, in _configure_calico
self.
File "./src/charm.py", line 305, in _configure_node
raise e
File "./src/charm.py", line 298, in _configure_node
node = self._calicoctl
File "./src/charm.py", line 640, in _calicoctl_get
output = self.calicoctl(
File "./src/charm.py", line 632, in calicoctl
return subprocess.
File "/usr/lib/
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/
raise CalledProcessEr
subprocess.
2024-04-26 16:00:11 DEBUG unit.calico/
Changed in charm-calico: | |
milestone: | none → 1.30+ck1 |
status: | New → Triaged |
summary: |
- One of calico instace stuck in "Configuring Calico" + One of calico instance stuck in "Configuring Calico" |
Changed in charm-calico: | |
milestone: | 1.30+ck1 → 1.30 |
Changed in charm-calico: | |
milestone: | 1.30+ck1 → 1.30+ck2 |
status: | Fix Committed → In Progress |
Having this issue in revision 105 on jammy. All of the calico nodes are stuck in this infinite loop failing to configure. If I ssh into any of the calico nodes and run the command (/opt/calicoctl /calicoctl get -o yaml --export node novel-bird it does not return null neither does it return a non-zero exit status of 1.
command: calicoctl get -o yaml --export node novel-bird
/opt/calicoctl/
output: org/v3 co.org/ kube-labels: '{"beta. kubernetes. io/arch" :"amd64" ,"beta. kubernetes. io/os": "linux" ,"juju- application" :"kubernetes- control- plane", "juju-charm" :"kubernetes- control- plane", "kubernetes. io/arch" :"amd64" ,"kubernetes. io/hostname" :"novel- bird"," kubernetes. io/os": "linux" ,"node- role.kubernetes .io/control- plane": ""}' tamp: null kubernetes. io/arch: amd64 kubernetes. io/os: linux application: kubernetes- control- plane control- plane io/arch: amd64 io/hostname: novel-bird io/os: linux role.kubernetes .io/control- plane: ""
apiVersion: projectcalico.
kind: Node
metadata:
annotations:
projectcali
creationTimes
labels:
beta.
beta.
juju-
juju-charm: kubernetes-
kubernetes.
kubernetes.
kubernetes.
node-
name: novel-bird
spec:
addresses:
- address: 192.168.2.38
type: InternalIP
orchRefs:
- nodeName: novel-bird
orchestrator: k8s
status: {}
error logs:
unit-calico-3: 09:10:37 ERROR unit.calico/ 3.juju- log b'resource does not exist: Node(novel-bird) with error: <nil>\n' 3.juju- log b'null\n' 3.juju- log Failed to configure node. juju/agents/ unit-calico- 3/charm/ ./src/charm. py", line 298, in _configure_node _get("node" , node_name) juju/agents/ unit-calico- 3/charm/ ./src/charm. py", line 640, in _calicoctl_get *args) juju/agents/ unit-calico- 3/charm/ ./src/charm. py", line 632, in calicoctl check_output( cmd, env=env, stderr= subprocess. PIPE, timeout=timeout) python3. 10/subprocess. py", line 421, in check_output python3. 10/subprocess. py", line 526, in run ror(retcode, process.args, CalledProcessEr ror: Command '['/opt/ calicoctl/ calicoctl' , 'get', '-o', 'yaml', '--export', 'node', 'novel-bird']' returned non-zero exit status 1. 3.juju- log Failed to configure Calico, will retry. juju/agents/ unit-calico- 3/charm/ ./src/charm. py", line 174, in _install_or_upgrade _configure_ calico( ) juju/agents/ unit-calico- 3/charm/ ./src/charm. py", line 125, in _configure_calico _configure_ node() juju/agents/ unit-calico- 3/charm/ ./src/charm. py", line 305, in _configure_node juju/agents/ unit-calico- 3/charm/ ./src/charm. py", line 298, in _configure_node _get("node" , node_name) juju/agents/ unit-calico- 3/charm/ ./src/charm. py", line 640, in _calicoctl_get
unit-calico-3: 09:10:37 ERROR unit.calico/
unit-calico-3: 09:10:37 ERROR unit.calico/
Traceback (most recent call last):
File "/var/lib/
node = self._calicoctl
File "/var/lib/
output = self.calicoctl(
File "/var/lib/
return subprocess.
File "/usr/lib/
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/
raise CalledProcessEr
subprocess.
unit-calico-3: 09:10:37 ERROR unit.calico/
Traceback (most recent call last):
File "/var/lib/
self.
File "/var/lib/
self.
File "/var/lib/
raise e
File "/var/lib/
node = self._calicoctl
File "/var/lib/
output = s...