charm release quincy/stable 214 is reporting missing python modules on install hook

[Andre Ruiz]

ceph-mon/0* error idle 0/lxd/0 172.27.76.109 hook failed: "install"
ceph-mon/1 error idle 1/lxd/0 172.27.76.106 hook failed: "install"
ceph-mon/2 error idle 2/lxd/0 172.27.76.105 hook failed: "install"

unit-ceph-mon-0: 01:22:18 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/ceph-mon-214 from API server
unit-ceph-mon-0: 01:22:19 INFO juju.worker.uniter hooks are retried true
unit-ceph-mon-0: 01:22:20 INFO juju.worker.uniter.storage initial storage attachments ready
unit-ceph-mon-0: 01:22:20 INFO juju.worker.uniter found queued "install" hook
unit-ceph-mon-0: 01:22:21 WARNING unit.ceph-mon/0.install Traceback (most recent call last):
unit-ceph-mon-0: 01:22:21 WARNING unit.ceph-mon/0.install File "/var/lib/juju/agents/unit-ceph-mon-0/charm/./src/charm.py", line 22, in <module>
unit-ceph-mon-0: 01:22:21 WARNING unit.ceph-mon/0.install import ceph_metrics
unit-ceph-mon-0: 01:22:21 WARNING unit.ceph-mon/0.install File "/var/lib/juju/agents/unit-ceph-mon-0/charm/src/ceph_metrics.py", line 22, in <module>
unit-ceph-mon-0: 01:22:21 WARNING unit.ceph-mon/0.install from charms.grafana_agent.v0 import cos_agent
unit-ceph-mon-0: 01:22:21 WARNING unit.ceph-mon/0.install File "/var/lib/juju/agents/unit-ceph-mon-0/charm/lib/charms/grafana_agent/v0/cos_agent.py", line 218, in <module>
unit-ceph-mon-0: 01:22:21 WARNING unit.ceph-mon/0.install import pydantic
unit-ceph-mon-0: 01:22:21 WARNING unit.ceph-mon/0.install ModuleNotFoundError: No module named 'pydantic'
unit-ceph-mon-0: 01:22:21 ERROR juju.worker.uniter.operation hook "install" (via hook dispatching script: dispatch) failed: exit status 1

If I enter the unit and install this module by hand, it works but then complains about another one ("cosl"). Network is ok, access to internet is ok, juju controller was re-installed, the same issue happens on 3 different machines.

I cut the bundle down to a single ceph-mon with no relations and it still happens. Seems like a bug was introduced in the charm released today.
This same bundle worked well a few days ago.

[Dan Hill (hillpd)]

I looked at this briefly with Andre tonight.

The `pydantic` dependency was added by a recent commit 3 days ago [0].

Two new dependencies were added to `test-requirements.txt` but they are not installed during deployment. I don't see them listed in `requirements.txt` but I did not have time to check external eggs.

[0] https://github.com/openstack/charm-ceph-mon/commit/6cd7be9036f251dfbe80dbb765570f2a2143eed0

[Andre Ruiz (andre-ruiz)]

I re-built the charm locally (git clone + charmcraft pack) with charmcraft v2.6 (which the changelog for that commit says is mandatory) and I saw all those dependencies being pulled into the charm, also verified inside the generated archive that they are in venv dir.

Installed that locally built charm but still same error, it cannot find them.

[Nobuto Murata (nobuto)]

Isn't this another example of the necessity of fixing the functional tests? We saw side effects of using charms from the charmstore instead of installing a charm with a proposed patch in the initial installation of the functional tests before.

https://opendev.org/openstack/charm-ceph-mon/src/branch/stable/quincy.2/tests/bundles/jammy-yoga.yaml#L76-L78
> ceph-mon:
> charm: ch:ceph-mon
> channel: quincy/edge

[Nobuto Murata (nobuto)]

Subscribing ~field-critical since the installation of the stable charm simply fails.

$ juju deploy ceph-mon --base ubuntu@22.04 --channel quincy/stable

[Peter Sabaini (peter-sabaini)]

This should be fixed by build update in https://github.com/openstack-charmers/charmed-openstack-info/pull/54 (rebuild is currently running)

[Felipe Reyes (freyes)]

stable/quincy.2 (the git branch used to built and publish to quincy/stable channel) is using charmcraft from 2.x/stable now, before it was using 2.0/stable

Related PR: https://github.com/openstack-charmers/charmed-openstack-info/pull/54

[Felipe Reyes (freyes)]

A new revision of the charm was built and published in Charmhub, revision 215 - https://launchpad.net/~openstack-charmers/charm-ceph-mon/+charm/charm-ceph-mon.stable-quincy.2.quincy/+build/24380

[Felipe Reyes (freyes)]

Marking quincy.2 task as fix released. And reef as invalid since that branch uses 2.x/stable to build.

[Peter Sabaini (peter-sabaini)]

@nobuto, wrt to functesting from reef onwards we're using a different release model with extra SQA release testing. We're looking into changing that for quincy as well.

[Andre Ruiz (andre-ruiz)]

I can confirm that release 215 is installing without those issues.

[Andre Ruiz (andre-ruiz)]

I'm wondering why I still had the issues when I built/packed the source locally using charmcraft 2.6 (which seemed to be the difference here). Did I clone the wrong sources? The ones I got are the ones indicated in the charmhub page as "Homepage" (link to [0]).

[0] - https://opendev.org/openstack/charm-ceph-mon

Just curious if that source link is wrong or the procedure to building from sources is broken in this case.

[Peter Sabaini (peter-sabaini)]

On 16.05.24 15:20, Andre Ruiz wrote:
> I'm wondering why I still had the issues when I built/packed the source
> locally using charmcraft 2.6 (which seemed to be the difference here).
> Did I clone the wrong sources? The ones I got are the ones indicated in
> the charmhub page as "Homepage" (link to [0]).
>
> [0] - https://opendev.org/openstack/charm-ceph-mon

That link is correct. Just to confirm, indeed the issue was that we didn't build the release version with charmcraft 2.6 (which is necessary to resolve the deps in the cos-agent library)

[Andre Ruiz (andre-ruiz)]

Ok, don't want to hijack the bug report and if this is an issue we can discuss separately, but it seems like something else is at play because I was not able to re-pack it locally with correct dependencies even using charmcraft 2.6. Maybe you can try on your local machine in a clean environment just to make sure.

Anyway, this seems to be resolved, thanks for the quick reaction.