k8s validation suite is incompatable with juju 3.1.1

[Alexander Balderson]

Running the validation suite a jammy machine running juju 3.1.1; the current version of pylibjuju is incompatible with the newest version of juju. an update to the library version should suffice to fix this.

Currently we're running on the 1.26 tag of the suite.

jobs/integration/test_service_endpoints.py::test_clusterip_service_endpoint
-------------------------------- live log setup --------------------------------
unknown facade SecretBackendsManager
unexpected facade SecretBackendsManager found, unable to decipher version to use
unknown facade SecretBackendsRotateWatcher
unexpected facade SecretBackendsRotateWatcher found, unable to decipher version to use
Task exception was never retrieved
future: <Task finished name='Task-18' coro=<Model._watch.<locals>._all_watcher() done, defined at /home/ubuntu/k8s-validation/.tox/py3/lib/python3.10/site-packages/juju/model.py:1189> exception=JujuAPIError('client incompatible with server 3.1.1')>
Traceback (most recent call last):
  File "/home/ubuntu/k8s-validation/.tox/py3/lib/python3.10/site-packages/juju/model.py", line 1192, in _all_watcher
    model_config = await self.get_config()
  File "/home/ubuntu/k8s-validation/.tox/py3/lib/python3.10/site-packages/juju/model.py", line 2182, in get_config
    result = await config_facade.ModelGet()
  File "/home/ubuntu/k8s-validation/.tox/py3/lib/python3.10/site-packages/juju/client/facade.py", line 481, in wrapper
    reply = await f(*args, **kwargs)
  File "/home/ubuntu/k8s-validation/.tox/py3/lib/python3.10/site-packages/juju/client/_client3.py", line 2142, in ModelGet
    reply = await self.rpc(msg)
  File "/home/ubuntu/k8s-validation/.tox/py3/lib/python3.10/site-packages/juju/client/facade.py", line 654, in rpc
    result = await self.connection.rpc(msg, encoder=TypeEncoder)
  File "/home/ubuntu/k8s-validation/.tox/py3/lib/python3.10/site-packages/juju/client/connection.py", line 638, in rpc
    raise errors.JujuAPIError(result)
juju.errors.JujuAPIError: client incompatible with server 3.1.1
ERROR: Got SIGTERM, handling it as a KeyboardInterrupt
ERROR: got KeyboardInterrupt signal
___________________________________ summary ____________________________________
ERROR: py3: keyboardinterrupt

[Bas de Bruijne (basdbruijne)]

The main branch is on juju==3.1.0.1 already, which is the latest version. We can switch to main for the juju 3.1.1 release testing.

[George Kraft (cynerva)]

It looks to me like there isn't yet a release of python-libjuju that supports juju 3.1.1. The latest release (3.1.0.1) was on Feb 10, but I see unreleased work in python-libjuju from 2 weeks ago[1] that adds support for the 3.1.1 facades.

For now, I'm guessing you might need to install python-libjuju directly from git/master to get something that works with 3.1.1.

I'll leave this issue open since we will need to update our dependency whenever the new version of python-libjuju is released.

[1]: https://github.com/juju/python-libjuju/pull/807

[Caner Derici (cderici)]

@George's comment is accurate. This will be fixed with a release with the new facades, we can certainly prioritize and get a new release going (we were already planning on doing it soon anyways). Have someone tried this with the master branch yet?

[Juan M. Tirado (tiradojm)]

Using the code in the master branch should work. That contains the latest facades. Said that, we're working on a new release with a more relaxed control of the facade versioning to avoid these situations.

[Caner Derici (cderici)]

There are some things to fix on the facade compatibility side on the pylibjuju, however, I don't seem to be able to reproduce this with pylibjuju release 3.1.0.1 against the juju 3.1.1

I just tried it again with the clean release version (from https://github.com/juju/python-libjuju/archive/refs/tags/3.1.0.1.tar.gz), and ran it against the juju controller 3.1.1 and although we're getting the same facade warnings, I'm not getting the "client incompatible" error that's mentioned. That's probably what lead us to think initially that the master branch would work because we weren't getting any errors with that.

Could you give more information, or double check that you're actually against 3.1.1, or maybe a link to the exact test that's running so I'll try to reproduce it?

Thanks!

[Juan M. Tirado (tiradojm)]

I ran an additional test using 3.1.0.1 with juju 3.2-beta1 which adds unknown facades and the operability can be maintained. See below:

 juju show-controller | grep agent-version
    agent-version: 3.2-beta1

```
from juju.model import Model

m = Model(); await m.connect()
unknown facade CrossModelSecrets
unexpected facade CrossModelSecrets found, unable to decipher version to use
unknown facade SecretBackendsManager
unexpected facade SecretBackendsManager found, unable to decipher version to use
unknown facade SecretBackendsRotateWatcher
unexpected facade SecretBackendsRotateWatcher found, unable to decipher version to use
unknown facade SecretsRevisionWatcher
unexpected facade SecretsRevisionWatcher found, unable to decipher version to use
This version was tested using 3.1.0 juju version 3.2-beta1 may have compatibility issues
This version was tested using 3.1.0 juju version 3.2-beta1 may have compatibility issues

a = await m.deploy("ubuntu")

status = await m.get_status()

status['applications']['ubuntu']['status']
<class 'juju.client._definitions.DetailedStatus'>({'data': {}, 'err': None, 'info': 'waiting for machine', 'kind': '', 'life': '', 'since': '2023-04-14T09:44:32.538357359Z', 'status': 'waiting', 'version': '', 'unknown_fields': {}})
```

When using a 3.1.2 controller (latest/stable) I can operate with python-libjuju 3.1.0.1.
```
juju status
Model Controller Cloud/Region Version SLA Timestamp
toremoveme test312 microk8s/localhost 3.1.2 unsupported 11:56:45+02:00

from juju.model import Model

m = Model(); await m.connect()
This version was tested using 3.1.1 juju version 3.1.2 may have compatibility issues
This version was tested using 3.1.1 juju version 3.1.2 may have compatibility issues
```

As mentioned by @cderici please, check the python-libjuju version that is currently being used and the controller you're connecting go. You can see the controller version in python-libjuju using:

```
from juju.controller import Controller

c = Controller(); await c.connect()
This version was tested using 3.1.1 juju version 3.1.2 may have compatibility issues

c.connection().info["server-version"]

```