start hook called after charm upgrade, no reboot

[Heather Lanigan]

In 2.8.4, upgrading a charm leads to the start hook being called instead of the upgrade-charm hook. The upgrade charm operation sets the Restart flag, to restart the uniter via the loops: ErrRestart.

unit-keystone-0: 12:56:45 INFO juju.worker.uniter reboot detected; triggering implicit start hook to notify charm

$ /snap/bin/juju show-status-log keystone/0
Time Type Status Message
29 Sep 2020 12:51:31Z workload waiting waiting for machine
29 Sep 2020 12:51:31Z juju-unit allocating
29 Sep 2020 12:52:45Z workload waiting installing agent
29 Sep 2020 12:52:46Z workload waiting agent initializing
29 Sep 2020 12:52:48Z workload maintenance installing charm software
29 Sep 2020 12:52:48Z juju-unit executing running install hook
29 Sep 2020 12:53:26Z workload maintenance Executing pre-install
29 Sep 2020 12:53:26Z workload maintenance Installing apt packages
29 Sep 2020 12:55:29Z workload maintenance Disabling unused apache configs
29 Sep 2020 12:55:31Z juju-unit executing running cluster-relation-created hook
29 Sep 2020 12:55:31Z juju-unit executing running leader-elected hook
29 Sep 2020 12:55:32Z workload blocked Missing relations: database, Allowed_units list provided but this unit not present
29 Sep 2020 12:55:33Z juju-unit executing running config-changed hook
29 Sep 2020 12:55:42Z workload maintenance Updating NRPE configuration
29 Sep 2020 12:55:44Z juju-unit executing running start hook
29 Sep 2020 12:55:46Z juju-unit idle <-- upgrade charm run here
29 Sep 2020 12:56:45Z juju-unit executing running start hook
29 Sep 2020 12:56:46Z workload blocked Missing relations: database, Allowed_units list provided but this unit not present
29 Sep 2020 12:56:47Z juju-unit executing running config-changed hook
29 Sep 2020 12:56:56Z workload maintenance Updating NRPE configuration

[Heather Lanigan (hmlanigan)]

https://github.com/juju/juju/pull/12069

[George Kraft (cynerva)]

This is breaking charm upgrades for people using Charmed Kubernetes: https://bugs.launchpad.net/charm-kubernetes-worker/+bug/1898549

We need this fix released ASAP.

[Cory Johns (johnsca)]

Just want to make clear that this bug means that the upgrade-charm hook is never run and no charm upgrades are working properly because of this.

[Kevin W Monroe (kwmonroe)]

For folks that are hitting this, the workaround noted in the related k8s bug is to manually run the upgrade-charm hook on failing applications, e.g.:

juju run --application etcd -- hooks/upgrade-charm