Upgrade fails with error importing PackageFinder

[Mike Wilson]

Just upgraded the charm to cs:~containers/docker-59. Errors on upgrade-charm hook:

unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm Traceback (most recent call last):
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm File "/var/lib/juju/agents/unit-docker-36/.venv/bin/pip", line 5, in <module>
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm from pip._internal import main
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm File "/var/lib/juju/agents/unit-docker-36/.venv/lib/python3.6/site-packages/pip/_internal/__init__.py", line 40, in <module>
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm from pip._internal.cli.autocompletion import autocomplete
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm File "/var/lib/juju/agents/unit-docker-36/.venv/lib/python3.6/site-packages/pip/_internal/cli/autocompletion.py", line 8, in <module>
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm from pip._internal.cli.main_parser import create_main_parser
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm File "/var/lib/juju/agents/unit-docker-36/.venv/lib/python3.6/site-packages/pip/_internal/cli/main_parser.py", line 12, in <module>
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm from pip._internal.commands import (
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm File "/var/lib/juju/agents/unit-docker-36/.venv/lib/python3.6/site-packages/pip/_internal/commands/__init__.py", line 6, in <module>
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm from pip._internal.commands.completion import CompletionCommand
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm File "/var/lib/juju/agents/unit-docker-36/.venv/lib/python3.6/site-packages/pip/_internal/commands/completion.py", line 6, in <module>
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm from pip._internal.cli.base_command import Command
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm File "/var/lib/juju/agents/unit-docker-36/.venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 23, in <module>
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm from pip._internal.index import PackageFinder
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm ImportError: cannot import name 'PackageFinder'
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm Traceback (most recent call last):
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm File "/var/lib/juju/agents/unit-docker-36/charm/hooks/upgrade-charm", line 8, in <module>
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm basic.bootstrap_charm_deps()
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm File "lib/charms/layer/basic.py", line 171, in bootstrap_charm_deps
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm 'pip'])
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm raise CalledProcessError(retcode, cmd)
unit-docker-36: 22:55:34 DEBUG unit.docker/36.upgrade-charm subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-docker-36/.venv/bin/pip', 'install', '-U', '--no-index', '-f', 'wheelhouse', 'pip']' returned non-zero exit status 1.
unit-docker-36: 22:55:34 ERROR juju.worker.uniter.operation hook "upgrade-charm" failed: exit status 1

[George Kraft (cynerva)]

Hey Mike.

What series are you running? We are aware of this issue occurring on Focal[1], but if this is occurring on Bionic or Xenial then this is a new case that we'll need to investigate.

[1]: https://bugs.launchpad.net/bugs/1870134

[George Kraft (cynerva)]

Looks like this might be related to the workaround used in https://bugs.launchpad.net/charm-docker/+bug/1862109.

[Mike Wilson (knobby)]

This is bionic still. I'm waiting for the upgrade path to look well-trodden before I try to move up.

[George Kraft (cynerva)]

Is this a unit that you previously ran `sudo /var/lib/juju/agents/unit-docker-20/.venv/bin/pip install --upgrade pip` on? If so, then that's what eventually led to this error condition. The next upgrade-charm would have "succeeded" but downgraded back to pip 18.x, causing future upgrade-charm hooks to fail.

You might be able to get out of this state by deleting the venv and forcing the charm to rebuild it:

juju run --application docker -- rm -rf ../.venv
juju run --application docker -- rm -rf wheelhouse/.bootstrapped
juju run --application docker -- hooks/upgrade-charm

[Mike Wilson (knobby)]

I think you are most likely right, but unfortunately, that didn't fix it.

unit-docker-97: 01:35:22 DEBUG unit.docker/97.upgrade-charm Processing ./wheelhouse/urllib3-1.25.8.tar.gz
unit-docker-97: 01:35:23 DEBUG unit.docker/97.upgrade-charm Complete output from command python setup.py egg_info:
unit-docker-97: 01:35:23 DEBUG unit.docker/97.upgrade-charm /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'project_urls'
unit-docker-97: 01:35:23 DEBUG unit.docker/97.upgrade-charm warnings.warn(msg)
unit-docker-97: 01:35:23 DEBUG unit.docker/97.upgrade-charm /usr/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'python_requires'
unit-docker-97: 01:35:23 DEBUG unit.docker/97.upgrade-charm warnings.warn(msg)
unit-docker-97: 01:35:23 DEBUG unit.docker/97.upgrade-charm error in urllib3 setup command: 'extras_require' must be a dictionary whose values are strings or lists of strings containing valid project/version requirement specifiers.
unit-docker-97: 01:35:23 DEBUG unit.docker/97.upgrade-charm
unit-docker-97: 01:35:23 DEBUG unit.docker/97.upgrade-charm ----------------------------------------

This is where I am stuck now.

[George Kraft (cynerva)]

Damn, okay. Looks like it's picking up libs from outside the venv, but I don't really know enough about layer-basic venv handling to give you any quick suggestions.

That error looks very similar to the one you originally reported in https://bugs.launchpad.net/charm-docker/+bug/1862109, so maybe you can manually upgrade setuptools in the venv again to get unstuck.

[Mike Wilson (knobby)]

I'm happy to do that, but I also understand the value of a reproduction environment. I'm happy to sit on this a little while if it could help. If not, I'll poke it like before.