Unable to juju upgrade-series from xenial to bionic without manual intervention

[Gareth Woolridge]

juju upgrade-series from xenial to bionic fails.

deploy a xenial instance:
juju deploy --series=xenial cs:ubuntu
juju deploy --series=xenial cs:canonical-livepatch
juju add-relation ubuntu canonical-livepatch

Upgrade series:
juju upgrade-series 0 prepare bionic
juju ssh 0
do-release-uprade

Post upgrade reboot, set complete to run the completion juju hooks:
juju upgrade-series 0 complete

This will hang running hooks, and juju status shows start hook errors:

canonical-livepatch/0* error idle 192.168.88.254 hook failed: "start"

juju logs show:
2020-06-15 15:42:41 ERROR juju.worker.uniter.operation runhook.go:136 hook "start" (via explicit, bespoke hook script) failed: signal: aborted (core dumped)
2020-06-15 15:42:52 ERROR juju.worker.uniter.operation runhook.go:136 hook "start" (via explicit, bespoke hook script) failed: signal: aborted (core dumped)
2020-06-15 15:43:13 ERROR juju.worker.uniter.operation runhook.go:136 hook "start" (via explicit, bespoke hook script) failed: signal: aborted (core dumped)
2020-06-15 15:43:55 ERROR juju.worker.uniter.operation runhook.go:136 hook "start" (via explicit, bespoke hook script) failed: signal: aborted (core dumped)

The workaround seems to be to nuke and recreate the venv:
moon127@hobbes:~$ juju run --application canonical-livepatch 'rm $JUJU_CHARM_DIR/wheelhouse/.bootstrapped'
moon127@hobbes:~$ juju run --application canonical-livepatch 'rm -rf $JUJU_CHARM_DIR/../.venv'
moon127@hobbes:~$ juju run --application canonical-livepatch 'hooks/update-status'
Reading package lists...
Building dependency tree...
Reading state information...
build-essential is already the newest version (12.4ubuntu1).
python3-setuptools is already the newest version (39.0.1-2).
python3-yaml is already the newest version (3.12-1build2).
python3-wheel is already the newest version (0.30.0-0.2).
python3-dev is already the newest version (3.6.7-1~18.04).
python3-pip is already the newest version (9.0.1-2.3~ubuntu1.18.04.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
virtualenv is already the newest version (15.1.0+ds-1.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /var/lib/juju/agents/unit-canonical-livepatch-0/.venv/bin/python3
Also creating executable in /var/lib/juju/agents/unit-canonical-livepatch-0/.venv/bin/python
Please make sure you remove any previous custom paths from your /root/.pydistutils.cfg file.
Installing setuptools, pkg_resources, pip, wheel...done.
Collecting pip
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-18.1
Looking in links: wheelhouse
Collecting setuptools
Collecting setuptools-scm
Installing collected packages: setuptools, setuptools-scm
  Found existing installation: setuptools 39.0.1
    Uninstalling setuptools-39.0.1:
      Successfully uninstalled setuptools-39.0.1
Successfully installed setuptools-41.6.0 setuptools-scm-1.17.0
Looking in links: wheelhouse
Processing ./wheelhouse/PyYAML-5.2.tar.gz
Processing ./wheelhouse/pyaml-19.12.0.tar.gz
Processing ./wheelhouse/Jinja2-2.11.1.tar.gz
Processing ./wheelhouse/setuptools_scm-1.17.0.tar.gz
Processing ./wheelhouse/Jinja2-2.10.1.tar.gz
Processing ./wheelhouse/MarkupSafe-1.1.1.tar.gz
Processing ./wheelhouse/Tempita-0.5.2.tar.gz
Processing ./wheelhouse/wheel-0.33.6.tar.gz
Processing ./wheelhouse/charmhelpers-0.20.10.tar.gz
Processing ./wheelhouse/PyYAML-5.3.tar.gz
Processing ./wheelhouse/charms.reactive-1.3.0.tar.gz
Processing ./wheelhouse/six-1.14.0.tar.gz
Processing ./wheelhouse/netaddr-0.7.19.tar.gz
Processing ./wheelhouse/setuptools-41.6.0.zip
Processing ./wheelhouse/pip-18.1.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
Collecting MarkupSafe>=0.23 (from Jinja2==2.11.1)
Collecting six (from charmhelpers==0.20.10)
Collecting netaddr (from charmhelpers==0.20.10)
Building wheels for collected packages: PyYAML, pyaml, Jinja2, MarkupSafe, setuptools-scm, Tempita, wheel, charmhelpers, netaddr, charms.reactive, setuptools, pip
  Running setup.py bdist_wheel for PyYAML: started
  Running setup.py bdist_wheel for PyYAML: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/da/bd/57/514d2fc1bb6c8d32b549a26b81b2ab01394a28e5e3d92f4080
  Running setup.py bdist_wheel for pyaml: started
  Running setup.py bdist_wheel for pyaml: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/a9/37/dd/7d287548dae60b2fc720451371eceebc366b0d5c6be5e1e53c
  Running setup.py bdist_wheel for Jinja2: started
  Running setup.py bdist_wheel for Jinja2: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/a4/3f/d5/68950124d9aa965c8e9dd726acfe759d7b63efffe5df548e55
  Running setup.py bdist_wheel for MarkupSafe: started
  Running setup.py bdist_wheel for MarkupSafe: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/9d/77/1e/d7ad426edab5e3558bdf275818b640da2d8f2b5b6871b564f3
  Running setup.py bdist_wheel for setuptools-scm: started
  Running setup.py bdist_wheel for setuptools-scm: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/fb/6c/4f/c69198cc0389d67e2c080bbb26b232affdbb004a1604258bc3
  Running setup.py bdist_wheel for Tempita: started
  Running setup.py bdist_wheel for Tempita: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/7b/2f/e9/7abd2fcd5c3462719081bf5f0801784b09fb1353ccddca3bd7
  Running setup.py bdist_wheel for wheel: started
  Running setup.py bdist_wheel for wheel: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/30/66/b2/5191c08eb573f22b7f4e762022861e1b1928fb797bc567dbf5
  Running setup.py bdist_wheel for charmhelpers: started
  Running setup.py bdist_wheel for charmhelpers: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/19/1a/b6/45084fd70f7bc135f4d9fce03498f07bf787939ee3b6b215be
  Running setup.py bdist_wheel for netaddr: started
  Running setup.py bdist_wheel for netaddr: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/63/01/d7/7908eff1f7761fdd0795e35ca57445cdea9d0ef147056856f4
  Running setup.py bdist_wheel for charms.reactive: started
  Running setup.py bdist_wheel for charms.reactive: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/a4/9f/f6/d02869572bc3c79cbb53ffccaf19ecaba68e68a55b291f2315
  Running setup.py bdist_wheel for setuptools: started
  Running setup.py bdist_wheel for setuptools: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/2e/6a/5a/932158ffc17ca188403dce2c7bc5aca4c5cbb1c2a9bf774a06
  Running setup.py bdist_wheel for pip: started
  Running setup.py bdist_wheel for pip: finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/29/0a/3d/7a3a59a6726c67480dc59e37e27cce89b3d6fbdbdd59150760
Successfully built PyYAML pyaml Jinja2 MarkupSafe setuptools-scm Tempita wheel charmhelpers netaddr charms.reactive setuptools pip
Installing collected packages: PyYAML, pyaml, MarkupSafe, Jinja2, setuptools-scm, Tempita, wheel, six, netaddr, charmhelpers, charms.reactive, setuptools, pip
Successfully installed Jinja2-2.11.1 MarkupSafe-1.1.1 PyYAML-5.2 Tempita-0.5.2 charmhelpers-0.20.10 charms.reactive-1.3.0 netaddr-0.7.19 pip-18.1 pyaml-19.12.0 setuptools-41.6.0 setuptools-scm-1.17.0 six-1.14.0 wheel-0.33.6
lxc
lxc
All snaps up to date.

[Zachary Zehring (zzehring)]

When hitting this issue, was the machine a LXD container? I was able to reproduce this issue when deploying to LXD, but not when deploying to a VM.

As an aside, I also experienced both ubuntu and canonical-livepatch units to hang on the start hook.

[Eric Chen (eric-chen)]

No further update over 1 year. Close this issue