Operator charm fails to install with `"meter-status-changed": charm missing from disk`

[Alexandre Gomes]

channel: latest/edge
revision: 58

When deploying the charm the landscape-server units get stuck in "(install) Installing apt packages".

The logs indicate `"meter-status-changed": charm missing from disk`. Full logs: https://pastebin.canonical.com/p/5WTwNJwKB7/

We managed to work around it by restarting jujud-machine agent and the charm seems to proceed correctly after that.

To deploy, we used this bundle[1] with this override[2].

[1] https://bazaar.launchpad.net/~sajoupa/canonical-mojo-specs/landscape-3/view/head:/landscape-internal/bundle.yaml
[2] https://bazaar.launchpad.net/~sajoupa/canonical-mojo-specs/landscape-3/view/head:/landscape-internal/ci/overlay.yaml

[Haw Loeung (hloeung)]

I don't think the issue is to do with the "charm missing from disk" as that's common if a hook isn't implemented. I think more work needs to be done to debug what's going on in legacy hooks/install. Maybe apt logs would help?

Anything in "ps" output at the time to see what processes were forked/spawned?

[Jeff Hillman (jhillman)]

I am having this same issue. This is the entirety of the unit log when it gets to this point.

2023-04-13 17:11:25 INFO juju unit_agent.go:289 Starting unit workers for "landscape-server/0"
2023-04-13 17:11:25 INFO juju.worker.apicaller connect.go:163 [ccba07] "unit-landscape-server-0" successfully connected to "10.141.15.103:17070"
2023-04-13 17:11:25 INFO juju.worker.apicaller connect.go:260 [ccba07] password changed for "unit-landscape-server-0"
2023-04-13 17:11:26 INFO juju.worker.apicaller connect.go:163 [ccba07] "unit-landscape-server-0" successfully connected to "10.141.15.128:17070"
2023-04-13 17:11:26 INFO juju.worker.migrationminion worker.go:142 migration phase is now: NONE
2023-04-13 17:11:26 INFO juju.worker.logger logger.go:120 logger worker started
2023-04-13 17:11:26 INFO juju.worker.upgrader upgrader.go:216 no waiter, upgrader is done
2023-04-13 17:11:26 ERROR juju.worker.meterstatus runner.go:93 error running "meter-status-changed": charm missing from disk
2023-04-13 17:11:26 INFO juju.worker.uniter uniter.go:326 unit "landscape-server/0" started
2023-04-13 17:11:26 INFO juju.worker.uniter uniter.go:634 resuming charm install
2023-04-13 17:11:26 INFO juju.worker.uniter.charm bundles.go:78 downloading ch:amd64/focal/landscape-server-87 from API server
2023-04-13 17:11:31 INFO juju.worker.uniter uniter.go:344 hooks are retried true
2023-04-13 17:11:31 INFO juju.worker.uniter.storage resolver.go:127 initial storage attachments ready
2023-04-13 17:11:31 INFO juju.worker.uniter resolver.go:159 found queued "install" hook
2023-04-13 17:11:31 INFO unit.landscape-server/0.juju-log server.go:316 Running legacy hooks/install.

This is the ps auxwww output (relevant part)

root 679 0.0 0.2 241032 9260 ? Ssl 17:10 0:00 /usr/lib/accountsservice/accounts-daemon
root 683 0.0 0.0 8540 2800 ? Ss 17:10 0:00 /usr/sbin/cron -f
message+ 687 0.5 0.1 7888 5164 ? Ss 17:10 0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root 691 0.0 0.0 81892 3732 ? Ssl 17:10 0:00 /usr/sbin/irqbalance --foreground
root 694 0.0 0.4 29868 18196 ? Ss 17:10 0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
root 695 0.0 0.2 236420 8988 ? Ssl 17:10 0:00 /usr/lib/policykit-1/polkitd --no-debug
root 702 0.4 0.9 948996 38396 ? Ssl 17:10 0:00 /usr/lib/snapd/snapd
root 703 0.1 0.1 17300 7644 ? Ss 17:10 0:00 /lib/systemd/systemd-logind
root 705 0.0 0.3 395472 13720 ? Ssl 17:10 0:00 /usr/lib/udisks2/udisksd
daemon 706 0.0 0.0 3796 2156 ? Ss 17:10 0:00 /usr/sbin/atd -f
root 759 0.0 0.3 245088 13512 ? Ssl 17:10 0:00 /usr/sbin/ModemManager
root 764 0.0 0.5 108120 20496 ? Ssl 17:10 0:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
root 778 0.0 0.0 7352 2416 ttyS0 Ss+ 17:10 0:00 /sbin/agetty -o -p -- \u --keep-baud 115200,38400,9600 ttyS0 vt220
root 779 0.0 0.0 0 ...