The charm remains stuck in "(install) Installing apt packages" state

[Laurent Sesquès]

In a fresh deploy, the landscape-server units remain stuck in the state:
landscape-server/0* maintenance executing 2 10.25.60.49 (install) Installing apt packages
landscape-server/1 maintenance executing 3 10.25.60.50 (install) Installing apt packages

The logs:
2022-10-27 13:25:21 INFO juju unit_agent.go:289 Starting unit workers for "landscape-server/0"
2022-10-27 13:25:21 WARNING juju.agent.setup agentconf.go:140 developer feature flags enabled: "image-metadata"
2022-10-27 13:25:21 INFO juju.worker.apicaller connect.go:163 [b1e438] "unit-landscape-server-0" successfully connected to "10.25.62.244:17070"
2022-10-27 13:25:21 INFO juju.worker.apicaller connect.go:260 [b1e438] password changed for "unit-landscape-server-0"
2022-10-27 13:25:21 INFO juju.worker.apicaller connect.go:163 [b1e438] "unit-landscape-server-0" successfully connected to "10.25.62.245:17070"
2022-10-27 13:25:21 INFO juju.worker.migrationminion worker.go:142 migration phase is now: NONE
2022-10-27 13:25:21 INFO juju.worker.logger logger.go:120 logger worker started
2022-10-27 13:25:21 INFO juju.worker.upgrader upgrader.go:216 no waiter, upgrader is done
2022-10-27 13:25:21 ERROR juju.worker.meterstatus runner.go:91 error running "meter-status-changed": charm missing from disk
2022-10-27 13:25:21 INFO juju.worker.uniter uniter.go:326 unit "landscape-server/0" started
2022-10-27 13:25:21 INFO juju.worker.uniter uniter.go:631 resuming charm install
2022-10-27 13:25:21 INFO juju.worker.uniter.charm bundles.go:78 downloading ch:amd64/focal/landscape-server-58 from API server
2022-10-27 13:25:26 INFO juju.worker.uniter uniter.go:344 hooks are retried true
2022-10-27 13:25:26 INFO juju.worker.uniter.storage resolver.go:127 initial storage attachments ready
2022-10-27 13:25:26 INFO juju.worker.uniter resolver.go:149 found queued "install" hook
2022-10-27 13:25:26 INFO unit.landscape-server/0.juju-log server.go:316 Running legacy hooks/install.

After restarting jujud-machine-X, operations resume:
2022-10-27 14:13:24 INFO juju unit_agent.go:289 Starting unit workers for "landscape-server/0"
2022-10-27 14:13:24 WARNING juju.agent.setup agentconf.go:140 developer feature flags enabled: "image-metadata"
2022-10-27 14:13:24 INFO juju.worker.apicaller connect.go:163 [b1e438] "unit-landscape-server-0" successfully connected to "10.25.62.245:17070"
2022-10-27 14:13:24 INFO juju.worker.apicaller connect.go:163 [b1e438] "unit-landscape-server-0" successfully connected to "10.25.62.246:17070"
2022-10-27 14:13:24 INFO juju.worker.migrationminion worker.go:142 migration phase is now: NONE
2022-10-27 14:13:24 INFO juju.worker.logger logger.go:120 logger worker started
2022-10-27 14:13:24 INFO juju.worker.upgrader upgrader.go:216 no waiter, upgrader is done
2022-10-27 14:13:24 INFO juju.worker.uniter uniter.go:326 unit "landscape-server/0" started
2022-10-27 14:13:24 INFO juju.worker.uniter uniter.go:344 hooks are retried true
2022-10-27 14:13:25 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "install" hook
2022-10-27 14:13:30 INFO juju.worker.uniter resolver.go:145 awaiting error resolution for "install" hook
2022-10-27 14:13:30 INFO unit.landscape-server/0.juju-log server.go:316 Running legacy hooks/install.
2022-10-27 14:13:41 INFO juju.worker.uniter.operation runhook.go:146 ran "install" hook (via hook dispatching script: dispatch)
2022-10-27 14:13:43 INFO juju.worker.uniter.operation runhook.go:146 ran "website-relation-created" hook (via hook dispatching script: dispatch)
2022-10-27 14:13:44 INFO juju.worker.uniter.operation runhook.go:146 ran "db-relation-created" hook (via hook dispatching script: dispatch)
2022-10-27 14:13:44 INFO juju.worker.uniter.operation runhook.go:146 ran "amqp-relation-created" hook (via hook dispatching script: dispatch)

[Haw Loeung (hloeung)]

Would be helpful if we had ps output at the time and/or /var/log/apt/history.log? I'm wondering if it's stuck at an apt prompt.

[Laurent Sesquès (sajoupa)]

This is definitely due to apt waiting for user input.
More precisely, it installs postfix as a dependency, and postfix asks some interactive questions.
This is all coming from lib/charms/operator_libs_linux/v0/apt.py.

[Mitch Burton (mitchburton)]

sajoupa has submitted a PR to fix this at the level of the operator lib: https://github.com/canonical/operator-libs-linux/pull/59