Containers are killed before any 'on stop/remove' handlers have a chance to run
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Fix Released
|
High
|
Thomas Miller |
Bug Description
When scaling down or removing units from a k8s charm, containers are killed almost immediately, leading to consistent errors and making it impossible to perform any cleanup actions on/with the container
This was found on juju 2.9.18 and micrk8s 1.21.6
This reliably leads to the following error on the charm container, which hangs around for longer:
```
unit-bar-0: 12:05:22 ERROR juju.worker.uniter pebble poll failed for container "bar": failed to get pebble info: cannot obtain system details: cannot communicate with server: Get "http://
```
This can be replicated by deploying, then removing the following charm:
```
#!/usr/bin/env python3
import logging
import os
import tenacity
import ops.charm
import ops.framework
import ops.main
import ops.model
import ops.pebble
logger = logging.
class FooCharm(
def __init__(self, *args):
def _on_bar_
container = event.workload
}
}
}
def _on_remove(self, _):
container = self.unit.
if __name__ == '__main__':
ops.
```
And will result in the traceback:
```
Traceback (most recent call last):
File "/usr/lib/
h.request(
File "/usr/lib/
self.
File "/usr/lib/
self.
File "/usr/lib/
self.
File "/usr/lib/
self.send(msg)
File "/usr/lib/
self.connect()
File "/var/lib/
self.
FileNotFoundError: [Errno 2] No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/
response = self.opener.
File "/usr/lib/
response = self._open(req, data)
File "/usr/lib/
result = self._call_
File "/usr/lib/
result = func(*args)
File "/var/lib/
return self.do_
File "/usr/lib/
raise URLError(err)
urllib.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./src/charm.py", line 52, in <module>
ops.
File "/var/lib/
_emit_
File "/var/lib/
event_
File "/var/lib/
framework.
File "/var/lib/
self.
File "/var/lib/
custom_
File "./src/charm.py", line 48, in _on_remove
container.
File "/var/lib/
self.
File "/var/lib/
return self._services_
File "/var/lib/
resp = self._request(
File "/var/lib/
response = self._request_
File "/var/lib/
raise ConnectionError
ops.pebble.
```
Changed in juju: | |
importance: | Undecided → High |
Changed in juju: | |
assignee: | nobody → Harry Pidcock (hpidcock) |
status: | New → In Progress |
Changed in juju: | |
milestone: | 2.9-next → 2.9.32 |
Changed in juju: | |
milestone: | 2.9.32 → 2.9.33 |
Changed in juju: | |
milestone: | 2.9.33 → 2.9.34 |
Changed in juju: | |
assignee: | Harry Pidcock (hpidcock) → Thomas Miller (tlmiller) |
Changed in juju: | |
milestone: | 2.9.34 → 2.9.35 |
Changed in juju: | |
milestone: | 2.9.35 → 2.9.36 |
Changed in juju: | |
milestone: | 2.9.36 → 2.9.37 |
Changed in juju: | |
milestone: | 2.9.37 → 2.9.38 |
Changed in juju: | |
milestone: | 2.9.38 → 2.9.39 |
Changed in juju: | |
milestone: | 2.9.39 → 2.9.40 |
Changed in juju: | |
milestone: | 2.9.40 → 2.9.41 |
Changed in juju: | |
milestone: | 2.9.41 → 2.9.42 |
Changed in juju: | |
milestone: | 2.9.42 → 2.9.43 |
Changed in juju: | |
milestone: | 2.9.43 → 2.9.44 |
Changed in juju: | |
milestone: | 2.9.44 → 2.9.43 |
status: | In Progress → Fix Committed |
Changed in juju: | |
status: | Fix Committed → Fix Released |
Partial fix https:/ /github. com/juju/ juju/pull/ 13816