[K8s] Install hook is not retriggered when Kubernetes Pod containing charm is recreated

[Claudiu Belu]

In Kubernetes, the Pods may be recreated due to various reasons (e.g.: Node restarted and the Pods have been recreated on other Nodes, Pods are evacuated from a Node, etc.). In the case of Kubernetes charms (using the operator framework), if a Pod is getting recreated, the install hook is not ran in the newly created Pod, which can cause some failures (e.g.: some dependencies may be installed in the install hook). After a bit of investigation, it seems that the upgrade hook is still triggered when recreating the Pod however.

To avoid this issue, ideally, the install hook would still be executed in case a Pod is recreated as it would be the most intuitive from a charm development perspective.

In the case in which dependencies are being installed in the install hook, a way to have them preinstalled would be useful (via charmcraft or similar), and it would speed up the charm deployment process.

How to reproduce this issue:

1. Have a charm with an install hook, in which an apt package is installed.
2. Assert that the apt package is installed in the container after deploying the charm.
3. Crash the charm's Kubernetes Pod, causing it to be recreated (the easiest way to do it is to delete the Pod: kubectl delete -n juju-model pod/charm-name-0
4. After the Pod is recreated, assert that the apt package no longer exists in the container.

Juju version: 3.2.4

[Ian Booth (wallyworld)]

This is an interesting topic. The hook sequencing was originally designed for machine deployments and indeed, the lifecycle of the charm was such that the install hook is by design only intended to be run once.

There may be a plausible argument to run the install hook once and only once when a pod is rescheduled. This would require work on Juju's side to reliably and unambiguously detect such a scenario, even if the reschedule happened when the controller might be down for some reason. And there'd be potential complications due to this being a change in behaviour and some charms may not have an idempotent install hook (even though they are supposed to) and so such a change would break existing charms.