Charms are failing basic hooks with ImportError after X-B Upgrade - Forgot to 'juju upgrade-series <m> prepare'

[Ryan Farrell]

After performing a Xenial to Bionic upgrad, a handful of charms have entered 'Error' state while running basic hooks:

nova-compute-kvm/30 active idle 32 10.55.65.18 Unit is ready
  ceilometer-agent/23 active idle 10.55.65.18 Unit is ready
  filebeat/30 error idle 10.55.65.18 hook failed: "update-status"
  landscape-client/14 active idle 10.55.65.18 System successfully registered
  lldpd/6 active idle 10.55.65.18 LLDP daemon running
  neutron-openvswitch/23 active idle 10.55.65.18 Unit is ready
  nrpe-host/21 active idle 10.55.65.18 icmp,5666/tcp ready
  ntp/25 error idle 10.55.65.18 123/udp hook failed: "update-status"
  rsyslog-forwarder-ha/31 unknown idle 10.55.65.18
  sysconfig-compute/3 error idle 10.55.65.18 hook failed: "update-status"
  telegraf/28 error idle 10.55.65.18 9103/tcp hook failed: "update-status"
  trilio-data-mover/109 error idle 10.55.65.18 hook failed: "update-status"

Upon reviewing the juju unit logs all charms are encountering an ImportError:

2020-03-18 13:32:04 DEBUG juju.worker.uniter.operation executor.go:59 running operation run update-status hook
2020-03-18 13:32:04 DEBUG juju.machinelock machinelock.go:156 acquire machine lock for uniter (run update-status hook)
2020-03-18 13:32:04 DEBUG juju.machinelock machinelock.go:166 machine lock acquired for uniter (run update-status hook)
2020-03-18 13:32:04 DEBUG juju.worker.uniter.operation executor.go:90 preparing operation "run update-status hook"
2020-03-18 13:32:04 DEBUG juju.worker.uniter.operation executor.go:90 executing operation "run update-status hook"
2020-03-18 13:32:04 DEBUG update-status Could not find platform independent libraries <prefix>
2020-03-18 13:32:04 DEBUG update-status Could not find platform dependent libraries <exec_prefix>
2020-03-18 13:32:04 DEBUG update-status Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
2020-03-18 13:32:04 DEBUG update-status Fatal Python error: Py_Initialize: Unable to get the locale encoding
2020-03-18 13:32:04 DEBUG update-status ImportError: No module named 'encodings'
2020-03-18 13:32:04 DEBUG update-status
2020-03-18 13:32:04 DEBUG update-status Current thread 0x00007fb437a56740 (most recent call first):
2020-03-18 13:32:05 ERROR juju.worker.uniter.operation runhook.go:132 hook "update-status" failed: signal: aborted (core dumped)

[Ryan Farrell (whereisrysmind)]

Note: these are all reactive charms and are having problems with their virtual environments

ubuntu@CMOOSCHSTUP7132:/var/lib/juju/agents$ source unit-ntp-25/.venv/
bin/ include/ lib/ share/
ubuntu@CMOOSCHSTUP7132:/var/lib/juju/agents$ source unit-ntp-25/.venv/bin/activate
(.venv) ubuntu@CMOOSCHSTUP7132:/var/lib/juju/agents$ python
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ImportError: No module named 'encodings'

Current thread 0x00007f68373e0740 (most recent call first):
Aborted (core dumped)

[Peter Sabaini (peter-sabaini)]

As the venv setup is handled by layer-basic, I wonder if the defect is really there. It looks like there is an invalid (old or incomplete) venv around.

I have tried to repro this but was unable to do so I'm afraid.

@Ryan: would you be able to provide more context on the issue?

- Steps to repro
- Logs or maybe even sosreports

Cheers!

[Ryan Farrell (whereisrysmind)]

This occurred when I ran a series upgrade, but forgot to prepare juju beforehand.

The solution involved the following workaround:

for CHARM_DIR in /var/lib/juju/agents/unit-ntp-25/charm \
/var/lib/juju/agents/unit-sysconfig-compute-3/charm \
/var/lib/juju/agents/unit-bcache-tuning-32/charm \
/var/lib/juju/agents/unit-hw-health-18/charm \
/var/lib/juju/agents/unit-trilio-data-mover-109/charm \
/var/lib/juju/agents/unit-telegraf-28/charm \
/var/lib/juju/agents/unit-filebeat-30/charm; do
echo $CHARM_DIR;
rm -rf $CHARM_DIR/wheelhouse/.bootstrapped $CHARM_DIR/../.venv;
cd $CHARMDIR;
JUJU_CHARM_DIR=$CHARM_DIR PYTHONPATH=$CHARM_DIR/lib python3 -c 'import charms.layer.basic; charms.layer.basic.bootstrap_charm_deps()';
done

[Launchpad Janitor (janitor)]

[Expired for Telegraf Charm because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for NTP Charm because there has been no activity for 60 days.]