runtime: unexpected return pc for github.com/juju/juju/upgrades.readAllStorageStateFiles called from 0x0

[Felipe Reyes]

[Impact]

An environment upgraded from 2.7.8 to 2.8.10 succeeded in all the units but one where jujud is crashing with no obvious reason, this unit is a lxd container running prometheus and few other subordinates.

According to my investigation the crash happens when jujud starts and tries to run the migration steps (github.com/juju/juju/upgrades/steps_28.go)

The stacktrace is available at https://pastebin.ubuntu.com/p/8jHTy7fYv3/

[John A Meinel (jameinel)]

Do you still have access to this machine? The actual failure is very strange, but I'm wondering if there is something about the agents configuration that we are reading and crashing on.

[Felipe Reyes (freyes)]

I don't have direct access to it, but I can ask BootStack to collect
the bits you need. What I already have available is a sosreport, you
can download it from
https://private-fileshare.canonical.com/~freyes/sosreport-prometheus-3-00306139-2021-05-12-veouqzq.tar.xz
it contains a copy of /var/lib/juju.

[John A Meinel (jameinel)]

just for some context, this is machine-10.log and the machine agent is the one that is failing to start

[Pen Gale (pengale)]

Per discussion in daily sync, the underlying issue may not be Juju's fault, but we shouldn't be dumping a stacktrace -- should be failing with something more useful.

[Harry Pidcock (hpidcock)]

Please let us know if this is still occurring, doesn't look like a juju issue, likely a go compiler issue or hardware issue.

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]