Integration tests that were previously working fail on 2.9.47

[Allan Vidal]

Steps to reproduce:
1. Clone this charm, which is just the template "kubernetes" charmcraft-generated charm + a few small modifications to make it run under Juju 2.9 (split charmcraft.yaml, pin juju==2.9.46.1 in tox.ini). Make sure to edit the workload image being referenced in metadata.yaml to something you can deploy:

    https://github.com/alnvdl-work/charm_2_9_47_bug

2. Bootstrap a new controller with Juju 2.9.47 targetting microk8s (stable/1.28).

3. Run `tox run -e integration` for the charm. It fails with:

   FAILED tests/integration/test_charm.py::test_build_and_deploy - juju.errors.JujuError: ['charm "local:jammy/tcharm-0" not found']

4. Destroy the previous controller and bootstrap a new one with Juju 2.9.46 (revision 25672, or snap revert can help), also targetting microk8s (stable/1.28).

5. Run `tox run -e integration` for that charm. It passes.

[Jack Shaw (jack-shaw)]

Hi, thanks for the report

Could you provide me with some more background around how you observed this?

I have replicated this with a machine controller

[Allan Vidal (alnvdl)]

Hi Jack,

I found this out because we have several small, relatively simple charms (4 at the moment, soon to be 7), and their integration tests all started failing our GitHub Actions workflows. The worked on Friday for some of the charms, but consistently failed on Monday for the exact same charm code.

These charms were originally based on the template kubernetes charm, so I suspected it was not our code, but something involving Juju or the test environment. After ruling out other changes in the GitHub runner test environment, I managed to reproduce the bug locally in my machine after upgrading to 2.9.47 and confirmed it by rolling back to 2.9.46.

The tests generated with the template are for demonstration purposes, but they are also fully functioning tests. They just don't make many assertions about what is deployed, but they exercise pretty much all of the basic setup.

So:
- charmcraft init --profile kubernetes
- Adapt charm to run on Juju 2.9.47 as I described above; if using my charm, just change the image tag to something you can deploy
- juju bootstrap on 2.9.47, pick microk8s as a cloud
- tox run -e integration --> it will fail

If you try the last two steps with 2.9.46, it works. If you had the charm installed before, you can just do `sudo snap revert juju` to go back to 2.9.46.

If you didn't have it installed, I guess you can try `sudo snap install juju --classic --revision=25672`, but I didn't try that.

This bug is affecting us quite a bit, as we need to target 2.9 for the IS-provided infra when deploying our internal services. For now, we are disabling the integration tests, but that's not ideal in the long term.

Allan

[John A Meinel (jameinel)]

Given that we had an API that was working to do deploy with a given parameter, while we updated the Juju CLI to be compatible with the API change, we shouldn't require that of all of our API clients.
I consider this a regression vs 2.9.46, so we should release a 2.9.48.

[Jack Shaw (jack-shaw)]

Fix here:
https://github.com/juju/juju/pull/17061