upgrade-hook failing with missing dependencies on wheelhouse.txt

[DUFOUR Olivier]

When upgrading the charm on an existing environment with a charm built from the current master branch. The neutron-api-plugin-ovn subordinate unit goes into an error during the upgrade hook complaining about 2 missing dependencies within the wheelhouse :
* hatchling
* calver

Complete error logs are attached to this ticket.

First error :
2023-05-15 03:43:57 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 ╰─> [3 lines of output]
2023-05-15 03:43:57 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 Looking in links: wheelhouse
2023-05-15 03:43:57 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 ERROR: Could not find a version that satisfies the requirement hatchling<2,>=1.6.0 (from versions: none)
2023-05-15 03:43:57 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 ERROR: No matching distribution found for hatchling<2,>=1.6.0
2023-05-15 03:43:57 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 [end of output]

Second error :
2023-05-15 04:02:23 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 × pip subprocess to install build dependencies did not run successfully.
2023-05-15 04:02:23 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 │ exit code: 1
2023-05-15 04:02:23 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 ╰─> [5 lines of output]
2023-05-15 04:02:23 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 Looking in links: wheelhouse
2023-05-15 04:02:23 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 Collecting setuptools
2023-05-15 04:02:23 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 Using cached setuptools-62.1.0-py3-none-any.whl
2023-05-15 04:02:23 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 ERROR: Could not find a version that satisfies the requirement calver (from versions: none)
2023-05-15 04:02:23 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 ERROR: No matching distribution found for calver
2023-05-15 04:02:23 WARNING unit.neutron-api-plugin-ovn/0.upgrade-charm logger.go:60 [end of output]

After adding them to src/wheelhouse.txt, rebuilding the charm with charmcraft and finally upgrading the charm, the upgrade hook works properly.

The testing environment is the following :
* Juju : 2.9.42
* Ubuntu : 22.04 / Jammy
* Openstack : Yoga
* Charmcraft : 2.2/stable
* Charm neutron-api-plugin-ovn channel before upgrade : yoga/stable
* Charm-neutron-api-plugin-ovn channel after upgrade: master

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ovn/+/883138

[DUFOUR Olivier (odufourc)]

Adding the complete Juju logs for one unit where the update-hook was failing.

It's a bit lengthy so the interesting timestamps are the following :
* 2023-05-12 06:16:40 --> deployment day with Yoga/stable branch on Jammy
* 2023-05-15 03:40:17 --> First attempt of upgrading local charm based on master branch
\--> fails with missing dependency with Hatchling
* 2023-05-15 03:58:56 --> Second attempt but adding hatchling as dependency in wheelhouse.txt
\--> fails with missing dependency with Calver
* 2023-05-15 04:09:56 --> Third attempt with calver added to wheelhouse.txt
\--> the upgrade hook is able to complete and juju status moves to active/idle.

In term of content for the wheelhouse, after adding hatchling and calver, I'm finding the following :
ubuntu@juju-b4eab2-0-lxd-6:/var/lib/juju/agents/unit-neutron-api-plugin-ovn-0/charm$ ls wheelhouse
Cython-0.29.34.tar.gz
Jinja2-3.1.2.tar.gz
MarkupSafe-2.1.2.tar.gz
PyYAML-6.0.tar.gz
calver-2022.6.26.tar.gz
certifi-2023.5.7.tar.gz
charmhelpers-1.1.1.dev72.zip
charms.openstack-0.0.1.dev1.zip
charms.reactive-1.5.2.tar.gz
charset-normalizer-3.1.0.tar.gz
debtcollector-2.5.0.tar.gz
dnspython-2.3.0.tar.gz
editables-0.3.tar.gz
flit_core-3.9.0.tar.gz
hatchling-1.17.0.tar.gz
idna-3.4.tar.gz
importlib_metadata-3.1.0.tar.gz
netaddr-0.7.19.tar.gz
netifaces-0.11.0.tar.gz
oslo.config-9.1.1.tar.gz
oslo.i18n-6.0.0.tar.gz
packaging-23.1.tar.gz
pathspec-0.11.1.tar.gz
pbr-5.11.1.tar.gz
pip-22.0.4.tar.gz
pluggy-1.0.0.tar.gz
poetry_core-1.6.0.tar.gz
psutil-5.9.5.tar.gz
pyaml-23.5.9.tar.gz
requests-2.30.0.tar.gz
rfc3986-2.0.0.tar.gz
setuptools-62.1.0.tar.gz
setuptools_scm-6.4.2.tar.gz
stevedore-5.0.0.tar.gz
tomli-2.0.1.tar.gz
trove-classifiers-2023.5.2.tar.gz
urllib3-2.0.2.tar.gz
wheel-0.40.0.tar.gz
wrapt-1.15.0.tar.gz
zipp-1.2.0.tar.gz

[Alex Kavanagh (ajkavanagh)]

I'm slightly concerned that the charm isn't picking up the dependencies at the charm build stage. Maybe we should switch this charm to a binary build (I think it is currently a source packages charm), and it might pick up the dependencies properly. Alternatively, if this is an issue with urllib3 having a major version bump [1], then perhaps this needs to solved in layer-basic so that all the charms can benefit?

[1] https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ovn/+/883138/comments/ccf31a01_209ad1bf

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Olivier DUFOUR-CUVILLIER <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ovn/+/883138
Reason: Abandoning patch in favor of adding the dependency on layer-basic

[DUFOUR Olivier (odufourc)]

I'm setting the previous patch as abandoned.
I have currently no knowledge about the procedure on how to move a charm to a binary build.

For urllib3 this would fix the issue with hachtling dependency but there would still be the calver dependency related to trove-classifiers. I'm unsure this would have its place in layer-basic.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ovn/+/883946

[Alex Kavanagh (ajkavanagh)]

HI Olivier, I've converted the charm to binary and it does now build and install on jammy (see https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ovn/+/883946). We're 'preferring' binary charms over source charms (i.e. building arch-specific wheels for reactive wheelhouse) as they tend to install quicker and all of the module build issues are resolved at build time, rather than charm install time (which leads to fewer install issues with reactive charms). Hope this helps.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ovn/+/883946
Committed: https://opendev.org/openstack/charm-neutron-api-plugin-ovn/commit/3fc51fadd8ca87534741be05a9ac350e1b8c2e56
Submitter: "Zuul (22348)"
Branch: master

commit 3fc51fadd8ca87534741be05a9ac350e1b8c2e56
Author: Alex Kavanagh <email address hidden>
Date: Tue May 23 15:44:35 2023 +0100

    Convert to binary reactive charm (source charm fails to install)

    Due to new install dependencies for urllib3, this charm has been
    converted to a binary charm so that the build dependencies are resolved
    at charm build, rather than charm install.

    Note: the 23.04 binary charm build fails due to missing support in
    charmcraft [1], but this should be addressed soon.

    [1] https://github.com/canonical/charmcraft/issues/1104

    Closes-Bug: #2019540
    Change-Id: I0834ddf7d63bf7f7d59a122bc07f007abffb5f3f

[DUFOUR Olivier (odufourc)]

Hello Alex,
Thank you for having taken care of this. This whole problematic and its solution were very insightful.