Clashing python package on upgrade

[Yanks]

When running `juju refresh`, juju unpacks the python packages in the venv/ directory. One of the packages `openstacksdk` uses Python stdlib `importlib` `metadata` module which tries to read the METADATA file to get the version string of the library.
However, since Juju unpacks the packages but not necessarily deletes the old ones, it leaves the venv directory with two versions of the openstack library: 2.1.0, 3.0.0
```
ubuntu@juju-d7b4aa-stg-github-runner-ps6-27:/var/lib/juju/agents/unit-stg-github-runne-0/charm$ echo venv/openstack*
venv/openstack venv/openstacksdk-2.1.0.dist-info venv/openstacksdk-3.0.0.dist-info
```
This made the charm error and the old library had to be manually deleted.

To reproduce:

$ juju deploy github-runner --channel=latest/edge --revision=130 --constraints="arch=amd64 cores=2 mem=16384M root-disk=69632M" --config path="canonical/github-runner-operator" --config token=<YOUR_GITHUB_TOKEN>--config virtual-machines=1 --config vm-cpu=2 --config vm-memory=8GiB --config vm-disk=40GiB --config runner-storage=juju-storage stg-github-runner
$ juju refresh stg-github-runner

[Tom Haddon (mthaddon)]

We experienced this again when going from a charm version that included openstacksdk 3.0.0 to 3.1.0.

[John A Meinel (jameinel)]

My understanding is that juju uses the manifest of files in the charm to determine what to delete from the old version and replace with the new version.
Is it possible there is some side-effect from the old version of the charm that creates additional content in the venv directory that is not represented by the charm itself?
(eg, is the charm running a pip install)

In general, juju intentionally leaves additional content alone, so that if a charm does drop content on disk, which we aren't tracking, we don't delete it. (one of the key files is the charm's .unit-state.db) (I believe we use the .zip file list, and then say 'what files in the new .zip aren't the same in the old .zip, remove the old, write the new.)