subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-ntp-14/.venv/bin/pip', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'pip']' returned non-zero exit status 1

[Felipe Reyes]

Upgrading ntp charm to 20.08 fails with the following stacktrace:

2020-08-20 18:38:39 DEBUG upgrade-charm Reading package lists...
2020-08-20 18:38:39 DEBUG upgrade-charm Building dependency tree...
2020-08-20 18:38:39 DEBUG upgrade-charm Reading state information...
2020-08-20 18:38:39 DEBUG upgrade-charm build-essential is already the newest version (12.4ubuntu1).
2020-08-20 18:38:39 DEBUG upgrade-charm python3-setuptools is already the newest version (39.0.1-2).
2020-08-20 18:38:39 DEBUG upgrade-charm python3-yaml is already the newest version (3.12-1build2).
2020-08-20 18:38:39 DEBUG upgrade-charm python3-wheel is already the newest version (0.30.0-0.2).
2020-08-20 18:38:39 DEBUG upgrade-charm python3-dev is already the newest version (3.6.7-1~18.04).
2020-08-20 18:38:39 DEBUG upgrade-charm python3-pip is already the newest version (9.0.1-2.3~ubuntu1.18.04.2).
2020-08-20 18:38:39 DEBUG upgrade-charm 0 upgraded, 0 newly installed, 0 to remove and 162 not upgraded.
2020-08-20 18:38:39 DEBUG upgrade-charm Reading package lists...
2020-08-20 18:38:40 DEBUG upgrade-charm Building dependency tree...
2020-08-20 18:38:40 DEBUG upgrade-charm Reading state information...
2020-08-20 18:38:40 DEBUG upgrade-charm python3-psutil is already the newest version (5.4.6-1build1~cloud0).
2020-08-20 18:38:40 DEBUG upgrade-charm 0 upgraded, 0 newly installed, 0 to remove and 162 not upgraded.
2020-08-20 18:38:41 DEBUG upgrade-charm Looking in links: wheelhouse
2020-08-20 18:38:41 DEBUG upgrade-charm Requirement already up-to-date: pip in /var/lib/juju/agents/unit-ntp-14/.venv/lib/python3.6/site-packages (18.1)
2020-08-20 18:38:41 DEBUG upgrade-charm Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/var/lib/juju/agents/unit-ntp-14/.venv/lib/python3.6/site-packages/Jinja2-2.10.3.dist-info/METADATA'
2020-08-20 18:38:41 DEBUG upgrade-charm
2020-08-20 18:38:42 DEBUG upgrade-charm Traceback (most recent call last):
2020-08-20 18:38:42 DEBUG upgrade-charm File "/var/lib/juju/agents/unit-ntp-14/charm/hooks/upgrade-charm", line 8, in <module>
2020-08-20 18:38:42 DEBUG upgrade-charm basic.bootstrap_charm_deps()
2020-08-20 18:38:42 DEBUG upgrade-charm File "lib/charms/layer/basic.py", line 197, in bootstrap_charm_deps
2020-08-20 18:38:42 DEBUG upgrade-charm _update_if_newer(pip, pre_install_pkgs)
2020-08-20 18:38:42 DEBUG upgrade-charm File "lib/charms/layer/basic.py", line 284, in _update_if_newer
2020-08-20 18:38:42 DEBUG upgrade-charm pkg], env=_get_subprocess_env())
2020-08-20 18:38:42 DEBUG upgrade-charm File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
2020-08-20 18:38:42 DEBUG upgrade-charm raise CalledProcessError(retcode, cmd)
2020-08-20 18:38:42 DEBUG upgrade-charm subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-ntp-14/.venv/bin/pip', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'pip']' returned non-zero exit status 1.
2020-08-20 18:38:42 ERROR juju.worker.uniter.operation runhook.go:132 hook "upgrade-charm" failed: exit status 1

This is a long standing environment, but this last upgrade was from ntp-40 to ntp-41.

[Felipe Reyes (freyes)]

Workaround:

juju ssh ntp/<UNIT> "sudo rm -rf /var/lib/juju/agents/unit-ntp-<UNIT>/.venv"
juju resolved ntp/<UNIT>

[Felipe Reyes (freyes)]

This seems to be kinda related to https://bugs.launchpad.net/charm-octavia/+bug/1891639 , and most likely this change[0] in layer-basic will fix this, although it will require a rebuild of the charm and publish it to the store.

[0] https://github.com/juju-solutions/layer-basic/pull/188

[Tom Haddon (mthaddon)]

New versions of the charm have been published since then. Can you confirm if this is still an issue?

[Nick DiLernia (ndilernia)]

Just ran into this on xenial/ussuri unit upgrading to NTP channel:stable rev:36