containerd stuck with "Container runtime not available"

Bug #1865543 reported by Joshua Genet
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Containerd Subordinate Charm
Fix Released
High
Joseph Borg

Bug Description

1/5 of our containerd units got blocked with the message "Container runtime not available".
This output is in all of the kubernetes unit logs but containerd/1 is the only unit that gets stuck in the blocked state.

---

The pod-logs contain the following:

pod-logs/kube-system-heapster-v1.6.0-beta.1-59878976bc-lcpqv-heapster.log:
E0301 02:30:05.021003 1 manager.go:101] Error in scraping containers from kubelet_summary:172.31.21.51:10250: request failed - "403 Forbidden", response: "Forbidden (user=system:serviceaccount:kube-system:heapster, verb=get, resource=nodes, subresource=stats)"

---

Here's a link to the artifacts/logs/bundles from our run:
https://oil-jenkins.canonical.com/artifacts/1e540251-9991-4bf8-aca5-f297417f629b/index.html

George Kraft (cynerva)
summary: - containerd failing to scrape containers from kubelet_summary
+ containerd stuck with "Container runtime not available"
Revision history for this message
George Kraft (cynerva) wrote :

containerd/1 is stuck in blocked status with "Container runtime not available". Looks like a status bug - I'm not seeing anything in the logs to indicate that containerd is actually failing, and it's pretty obvious looking at the charm code[1] that the status isn't being re-checked during the update-status hook like it should be.

[1]: https://github.com/charmed-kubernetes/charm-containerd/blob/d38de1f01dd0a4d686c0930062744228a332af38/reactive/containerd.py#L326-L331

Changed in charm-containerd:
status: New → Confirmed
Joseph Borg (joeborg)
Changed in charm-containerd:
assignee: nobody → Joseph Borg (joeborg)
Joseph Borg (joeborg)
Changed in charm-containerd:
importance: Undecided → High
status: Confirmed → In Progress
Revision history for this message
Joseph Borg (joeborg) wrote :
Joseph Borg (joeborg)
tags: added: review-needed
George Kraft (cynerva)
Changed in charm-containerd:
status: In Progress → Fix Committed
Joseph Borg (joeborg)
tags: removed: review-needed
Changed in charm-containerd:
milestone: none → 1.17+ck3
milestone: 1.17+ck3 → 1.18
Changed in charm-containerd:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.