Charm code upgraded without `upgrade-charm` event if refresh while unit in error status
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Triaged
|
High
|
Alastair Flynn |
Bug Description
If a unit is in error status (charm raised uncaught exception) before the user runs `juju refresh`, that unit will not get an upgrade-charm event, but its code will be upgraded.
Impact: if the user `juju refresh`es to a charm revision that contains a bug or if they accidentally selected the wrong revision, they will not be able to rollback safely
Steps to reproduce:
1. git clone https:/
2. cd bug-report-
3. charmcraft pack
4. On LXD controller, juju add-model foo1
5. juju deploy ./foo_ubuntu-
6. juju ssh foo/1
7. Edit /var/lib/
8. (exit from unit ssh)
9. jhack ffwd
- or wait for update-status
10. Unit 1 should be in error status
11. touch src/bar
12. charmcraft pack
13. juju refresh foo --path ./foo_ubuntu-
14. Wait a few minutes for foo/1 update-status to retry & exit error status
15. ls /var/lib/
16. Look at juju debug-log
versions:
juju agent: 3.4.2
```
$ snap list
Name Version Rev Tracking Publisher Notes
charmcraft 2.6.0 3451 latest/stable canonical✓ classic
core18 20240416 2823 latest/stable canonical✓ base
core20 20240416 2318 latest/stable canonical✓ base
core22 20240408 1380 latest/stable canonical✓ base
google-cloud-cli 478.0.0 243 latest/stable/… google-cloud-sdk✓ classic
juju 3.4.2 26968 3.4/stable canonical✓ -
lxd 5.0.3-d921d2e 28373 5.0/stable/… canonical✓ -
microk8s v1.29.5 6884 1.29-strict/stable canonical✓ -
snapd 2.63 21759 latest/stable canonical✓ snapd
```
Expected behavior:
After juju stops retrying the failed event on foo/1, it upgrades the charm code and emits an upgrade-charm event. Then it emits a config-changed event and then a leader-
Actual behavior:
The charm is upgraded but no upgrade-charm event is emitted. Then a leader-
Juju debug-log after `juju refresh` shows:
```
machine-2: 14:26:02 INFO juju.downloader downloading from local:jammy/foo-1
unit-foo-2: 14:26:02 INFO juju.worker.
machine-0: 14:26:02 INFO juju.downloader downloading from local:jammy/foo-1
unit-foo-0: 14:26:02 INFO juju.worker.
machine-0: 14:26:02 INFO juju.downloader download complete ("local:
machine-2: 14:26:02 INFO juju.downloader download complete ("local:
machine-0: 14:26:02 INFO juju.downloader download verified ("local:
machine-2: 14:26:02 INFO juju.downloader download verified ("local:
unit-foo-0: 14:26:05 INFO juju.worker.uniter found queued "upgrade-charm" hook
unit-foo-2: 14:26:05 INFO juju.worker.uniter found queued "upgrade-charm" hook
unit-foo-0: 14:26:05 DEBUG unit.foo/0.juju-log ops 2.14.0 up and running.
unit-foo-0: 14:26:05 INFO unit.foo/0.juju-log Running legacy hooks/upgrade-
unit-foo-2: 14:26:05 DEBUG unit.foo/2.juju-log ops 2.14.0 up and running.
unit-foo-2: 14:26:05 INFO unit.foo/2.juju-log Running legacy hooks/upgrade-
unit-foo-0: 14:26:05 DEBUG unit.foo/0.juju-log ops 2.14.0 up and running.
unit-foo-0: 14:26:05 DEBUG unit.foo/0.juju-log Charm called itself via hooks/upgrade-
unit-foo-2: 14:26:05 DEBUG unit.foo/2.juju-log ops 2.14.0 up and running.
unit-foo-0: 14:26:05 DEBUG unit.foo/0.juju-log Legacy hooks/upgrade-charm exited with status 0.
unit-foo-2: 14:26:05 DEBUG unit.foo/2.juju-log Charm called itself via hooks/upgrade-
unit-foo-0: 14:26:05 DEBUG unit.foo/0.juju-log Emitting Juju event upgrade_charm.
unit-foo-0: 14:26:05 INFO unit.foo/0.juju-log Event: <class 'ops.charm.
unit-foo-2: 14:26:05 DEBUG unit.foo/2.juju-log Legacy hooks/upgrade-charm exited with status 0.
unit-foo-2: 14:26:05 DEBUG unit.foo/2.juju-log Emitting Juju event upgrade_charm.
unit-foo-2: 14:26:05 INFO unit.foo/2.juju-log Event: <class 'ops.charm.
unit-foo-0: 14:26:05 INFO juju.worker.
unit-foo-0: 14:26:05 INFO juju.worker.uniter found queued "config-changed" hook
unit-foo-2: 14:26:05 INFO juju.worker.
unit-foo-2: 14:26:05 INFO juju.worker.uniter found queued "config-changed" hook
unit-foo-0: 14:26:06 DEBUG unit.foo/0.juju-log ops 2.14.0 up and running.
unit-foo-0: 14:26:06 DEBUG unit.foo/0.juju-log Emitting Juju event config_changed.
unit-foo-0: 14:26:06 INFO unit.foo/0.juju-log Event: <class 'ops.charm.
unit-foo-2: 14:26:06 DEBUG unit.foo/2.juju-log ops 2.14.0 up and running.
unit-foo-2: 14:26:06 DEBUG unit.foo/2.juju-log Emitting Juju event config_changed.
unit-foo-2: 14:26:06 INFO unit.foo/2.juju-log Event: <class 'ops.charm.
unit-foo-0: 14:26:06 INFO juju.worker.
unit-foo-2: 14:26:06 INFO juju.worker.
unit-foo-0: 14:26:06 DEBUG unit.foo/0.juju-log ops 2.14.0 up and running.
unit-foo-0: 14:26:06 DEBUG unit.foo/0.juju-log Emitting Juju event leader_
unit-foo-0: 14:26:06 INFO unit.foo/0.juju-log Event: <class 'ops.charm.
unit-foo-0: 14:26:06 INFO juju.worker.
unit-foo-1: 14:26:24 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-foo-1: 14:26:24 DEBUG unit.foo/1.juju-log ops 2.14.0 up and running.
unit-foo-1: 14:26:24 DEBUG unit.foo/1.juju-log Emitting Juju event update_status.
unit-foo-1: 14:26:24 INFO unit.foo/1.juju-log Event: <class 'ops.charm.
unit-foo-1: 14:26:24 ERROR unit.foo/1.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/
ops.
File "/var/lib/
return main(charm_class, use_juju_
File "/var/lib/
manager.run()
File "/var/lib/
self._emit()
File "/var/lib/
_emit_
File "/var/lib/
event_
File "/var/lib/
framework.
File "/var/lib/
self.
File "/var/lib/
custom_
File "/var/lib/
raise Exception
Exception
unit-foo-1: 14:26:24 ERROR juju.worker.
unit-foo-1: 14:26:24 INFO juju.worker.
machine-1: 14:26:24 INFO juju.downloader downloading from local:jammy/foo-1
machine-1: 14:26:24 INFO juju.downloader download complete ("local:
machine-1: 14:26:24 INFO juju.downloader download verified ("local:
unit-foo-1: 14:26:27 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-foo-1: 14:27:48 INFO juju.worker.uniter awaiting error resolution for "update-status" hook
unit-foo-1: 14:27:48 DEBUG unit.foo/1.juju-log ops 2.14.0 up and running.
unit-foo-1: 14:27:48 DEBUG unit.foo/1.juju-log Emitting Juju event update_status.
unit-foo-1: 14:27:48 INFO unit.foo/1.juju-log Event: <class 'ops.charm.
unit-foo-1: 14:27:48 INFO juju.worker.
unit-foo-1: 14:27:49 DEBUG unit.foo/1.juju-log ops 2.14.0 up and running.
unit-foo-1: 14:27:49 DEBUG unit.foo/1.juju-log Emitting Juju event leader_
unit-foo-1: 14:27:49 INFO unit.foo/1.juju-log Event: <class 'ops.charm.
unit-foo-1: 14:27:49 INFO juju.worker.
unit-foo-1: 14:27:49 DEBUG unit.foo/1.juju-log ops 2.14.0 up and running.
unit-foo-1: 14:27:49 DEBUG unit.foo/1.juju-log Emitting Juju event config_changed.
unit-foo-1: 14:27:49 INFO unit.foo/1.juju-log Event: <class 'ops.charm.
unit-foo-1: 14:27:49 INFO juju.worker.
```
Note that unit 1 does not receive upgrade-charm event, and that leader-
Changed in juju: | |
milestone: | 3.4.5 → 3.4.6 |
Changed in juju: | |
milestone: | 3.4.6 → 3.4.7 |
Changed in juju: | |
milestone: | 3.4.7 → 3.5.6 |
Additional context: https:/ /matrix. to/#/!xzmWHtGpP fVCXKivIh: ubuntu. com/$6DnO9iXzFN FZDj1sFTOPeamP8 Ez5xAwPab4COlp8 _I8?via= ubuntu. com&via= matrix. org&via= fsfe.org