python-libjuju doesn't populate the 'charm' field from subordinates in get_status()

[Felipe Reyes]

The zaza testing software has had a function named get_subordinates() which relies on finding the charm's name (not the app name) within the 'subordinates' tree, more details of the implementation can be found at https://github.com/openstack-charmers/zaza/blob/master/zaza/utilities/juju.py#L445

Here there is a test case that uses libjuju:

Steps:

juju add-model mymodel
juju deploy --series focal ch:ubuntu
juju deploy --series focal ch:ntp
juju add-relation ubuntu ntp
juju wait
cat << EOF > my.py
import asyncio
import os

from juju.model import Model

async def main():
    model = Model()
    await model.connect_model(os.environ['JUJU_MODEL'])
    status = await model.get_status()
    print(status.applications['ubuntu']['units']['ubuntu/0']['subordinates'])
    print(status.applications['ntp']['units'])
    assert status.applications['ubuntu']['units']['ubuntu/0']['subordinates']['ntp/0']['charm'], "The charm field shouldn't be empty"

if __name__ == "__main__":
    asyncio.run(main())
EOF
JUJU_MODEL=mymodel python3 my.py

Expected result:

the 'charm' field available at status.applications['ubuntu']['units']['ubuntu/0']['subordinates']['ntp/0']['charm'] contains the charm used to deploy that unit (e.g. ch:amd64/focal/ntp-49 ).

Actual result:

the 'charm' field is empty and only set at the same level of the units key (e.g. status['ntp']['charm'])

[Felipe Reyes (freyes)]

my environment is running juju 2.9.34

the issue is reproducible in 2.9.33 too:
https://github.com/openstack-charmers/zaza/runs/7955966658?check_suite_focus=true#step:4:831
https://github.com/openstack-charmers/zaza/runs/7955966658?check_suite_focus=true#step:5:24

[Felipe Reyes (freyes)]

This seems to be a juju issue and not exclusively a libjuju since the test case passes when using 2.8

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
default localhost-localhost localhost/localhost 2.8.13 unsupported 18:41:52Z

App Version Status Scale Charm Store Rev OS Notes
ntp 3.5 active 1 ntp jujucharms 49 ubuntu
ubuntu 20.04 active 1 ubuntu jujucharms 20 ubuntu

Unit Workload Agent Machine Public address Ports Message
ubuntu/0* active idle 0 10.74.32.11
  ntp/0* active idle 10.74.32.11 123/udp chrony: Ready, time sync disabled in container

Machine State DNS Inst id Series AZ Message
0 started 10.74.32.11 juju-c8f53f-0 focal Running

$ cat my.py
import asyncio
import os

from juju.model import Model

async def main():
    model = Model()
    await model.connect_model(os.environ['JUJU_MODEL'])
    status = await model.get_status()
    print(status.applications['ubuntu']['units']['ubuntu/0']['subordinates'])
    print(status.applications['ntp']['units'])
    print(status.applications['ntp']['charm'])
    print(status.applications['ubuntu']['charm'])
    print("status.applications['ubuntu']['units']['ubuntu/0']['subordinates']['ntp/0']['charm'] -> ", status.applications['ubuntu']['units']['ubuntu/0']['subordinates']['ntp/0']['charm'])

if __name__ == "__main__":
    asyncio.run(main())

$ JUJU_MODEL=default ./venv/bin/python3 my.py
{'ntp/0': <class 'juju.client._definitions.UnitStatus'>({'address': None, 'agent_status': <class 'juju.client._definitions.DetailedStatus'>({'data': {}, 'err': None, 'info': '', 'kind': '', 'life': '', 'since': '2022-08-22T18:29:45.486984344Z', 'status': 'idle', 'version': '2.8.13', 'unknown_fields': {}}), 'charm': 'cs:ntp-49', 'leader': True, 'machine': '', 'opened_ports': ['123/udp'], 'provider_id': None, 'public_address': '10.74.32.11', 'subordinates': {}, 'workload_status': <class 'juju.client._definitions.DetailedStatus'>({'data': {}, 'err': None, 'info': 'chrony: Ready, time sync disabled in container', 'kind': '', 'life': '', 'since': '2022-08-22T18:29:40.017088315Z', 'status': 'active', 'version': '', 'unknown_fields': {}}), 'workload_version': '3.5', 'unknown_fields': {}})}
{}
cs:ntp-49
cs:ubuntu-20
status.applications['ubuntu']['units']['ubuntu/0']['subordinates']['ntp/0']['charm'] -> cs:ntp-49

[Felipe Reyes (freyes)]

Here we have a test that's exercising get_subordinates()[0] and passing on juju 2.8, but failing on juju 2.9

juju 2.8 - https://github.com/openstack-charmers/zaza/runs/7964141009?check_suite_focus=true
juju 2.9 - https://github.com/openstack-charmers/zaza/runs/7964140964?check_suite_focus=true#step:4:842

[0] https://github.com/openstack-charmers/zaza/pull/556/commits/8f0dcf4f4862ab786ebb61327e883caecc424f0e#diff-f739ef17df6a8889195338692cdf40f52eb69eebd9f3d2b1d31e15c98d0d0bba

[Juan M. Tirado (tiradojm)]

What version of Pylibjuju are you using?

[Felipe Reyes (freyes)]

we pin libjuju to "<3.0" - https://github.com/openstack-charmers/zaza/blob/master/requirements.txt#L5 - so my environment 2.9.11

$ .tox/func-target/bin/pip3 list --local | grep juju
juju 2.9.11

[Felipe Reyes (freyes)]

Testing with libjuju 3.0.1 gives the same result.

 /tmp $ python3 -m venv venv-juju3
 /tmp $ ./venv-juju3/bin/pip
pip pip3 pip3.10
 /tmp $ ./venv-juju3/bin/pip3 install juju
Collecting juju
  Using cached juju-3.0.1-py3-none-any.whl
...
Installing collected packages: toposort, pytz, pyasn1, mypy-extensions, websockets, websocket-client, urllib3, typing-extensions, six, rsa, pyyaml, pyRFC3339, pycparser, pyasn1-modules, protobuf, oauthlib, idna, charset-normalizer, certifi, cachetools, typing-inspect, requests, python-dateutil, jujubundlelib, google-auth, cffi, requests-oauthlib, PyNaCl, cryptography, bcrypt, pymacaroons, paramiko, kubernetes, macaroonbakery, theblues, juju
Successfully installed PyNaCl-1.5.0 bcrypt-3.2.2 cachetools-5.2.0 certifi-2022.6.15 cffi-1.15.1 charset-normalizer-2.1.1 cryptography-37.0.4 google-auth-2.11.0 idna-3.3 juju-3.0.1 jujubundlelib-0.5.7 kubernetes-24.2.0 macaroonbakery-1.3.1 mypy-extensions-0.4.3 oauthlib-3.2.0 paramiko-2.11.0 protobuf-3.20.1 pyRFC3339-1.1 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.21 pymacaroons-0.13.0 python-dateutil-2.8.2 pytz-2022.2.1 pyyaml-6.0 requests-2.28.1 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 theblues-0.5.2 toposort-1.7 typing-extensions-4.3.0 typing-inspect-0.8.0 urllib3-1.26.12 websocket-client-1.3.3 websockets-10.3
 /tmp $ ./venv-juju3/bin/pip3 list --local | grep juju
juju 3.0.1
jujubundlelib 0.5.7
 /tmp $ juju status
Model Controller Cloud/Region Version SLA Timestamp
mymodel lxd-cluster-default lxd-cluster/default 2.9.34.1 unsupported 17:20:15-04:00

App Version Status Scale Charm Channel Rev Exposed Message
ntp 3.5 active 1 ntp stable 49 no chrony: Ready, time sync disabled in container
ubuntu 20.04 active 1 ubuntu stable 20 no

Unit Workload Agent Machine Public address Ports Message
ubuntu/0* active idle 0 192.168.11.210
  ntp/0* active idle 192.168.11.210 123/udp chrony: Ready, time sync disabled in container

Machine State Address Inst id Series AZ Message
0 started 192.168.11.210 juju-53bfb6-0 focal Running

 /tmp $ JUJU_MODEL=mymodel ./venv-juju3/bin/python3 my.py
{'ntp/0': <class 'juju.client.old_clients._definitions.UnitStatus'>({'address': None, 'agent_status': <class 'juju.client.old_clients._definitions.DetailedStatus'>({'data': {}, 'err': None, 'info': '', 'kind': '', 'life': '', 'since': '2022-08-22T17:47:17.452652569Z', 'status': 'idle', 'version': '2.9.34.1', 'unknown_fields': {}}), 'charm': '', 'leader': True, 'machine': '', 'opened_ports': ['123/udp'], 'provider_id': None, 'public_address': '192.168.11.210', 'subordinates': {}, 'workload_status': <class 'juju.client.old_clients._definitions.DetailedStatus'>({'data': {}, 'err': None, 'info': 'chrony: Ready, time sync disabled in container', 'kind': '', 'life': '', 'since': '2022-08-22T17:40:13.637132826Z', 'status': 'active', 'version': '', 'unknown_fields': {}}), 'workload_version': '3.5', 'unknown_fields': {}})}
{}
Traceback (most recent call last):
  File "/tmp/my.py", line 15, in ...

[Felipe Reyes (freyes)]

We came up with a workaround to get the information we need from the status, although we are keeping it behind a feature flag - https://github.com/openstack-charmers/zaza/commit/fc268181e8da1cb4293328fa2e78082873d98697

[Caner Derici (cderici)]

Yeah I'm able to reproduce this, and can confirm that the 'charm' field is missing. Interestingly, that seems to be the only field that's missing there, all the others are the same. That should make it easy to zero in on what's going on.

And I agree, this is about Juju, and not libjuju (libjuju is just calling and returning the result of `ClientFacade.FullStatus()`. And I don't think the libjuju version will matter, since this is before 3.0.

The only difference is that the best ClientFacade version for Juju v2.8 is v3 and for Juju 2.9 it's v5, which I think what's causing the discrepancy here.

For `status.applications['ubuntu']['units']['ubuntu/0']['subordinates']['ntp/0']`

This is the one Felipe sees with Juju 2.8 (I didn't have the chance to run it myself):

{'address': None,
'agent_status': <class 'juju.client._definitions.DetailedStatus'>({'data': {}, 'err': None, 'info': '', 'kind': '', 'life': '', 'since': '2022-08-22T18:29:45.486984344Z', 'status': 'idle', 'version': '2.8.13', 'unknown_fields': {}}),
'charm': 'cs:ntp-49',
'leader': True,
'machine': '',
'opened_ports': ['123/udp'],
'provider_id': None,
'public_address': '10.74.32.11',
'subordinates': {},
'workload_status': <class 'juju.client._definitions.DetailedStatus'>({'data': {}, 'err': None, 'info': 'chrony: Ready, time sync disabled in container', 'kind': '', 'life': '', 'since': '2022-08-22T18:29:40.017088315Z', 'status': 'active', 'version': '', 'unknown_fields': {}}),
'workload_version': '3.5', 'unknown_fields': {}}

And this is the one that I see with Juju 2.9:

{'address': None,
'agent_status': <class 'juju.client._definitions.DetailedStatus'>({'data': {}, 'err': None, 'info': '', 'kind': '', 'life': '', 'since': '2022-08-23T23:04:43.396538679Z', 'status': 'idle', 'version': '2.9.34.1', 'unknown_fields': {}}),
'charm': '', <-------------------------------------------- This is the only difference
'leader': True,
'machine': '',
'opened_ports': ['123/udp'],
'provider_id': None,
'public_address': '10.42.51.244',
'subordinates': {},
'workload_status': <class 'juju.client._definitions.DetailedStatus'>({'data': {}, 'err': None, 'info': 'chrony: Ready, time sync disabled in container', 'kind': '', 'life': '', 'since': '2022-08-23T23:04:39.921092382Z', 'status': 'active', 'version': '', 'unknown_fields': {}}),
'workload_version': '3.5', 'unknown_fields': {}}

[Juan M. Tirado (tiradojm)]

I could reproduce this issue using pylibjuju and a 2.9.33 controller. Information inside the MongoDB looks correct.

[Caner Derici (cderici)]

Fix is under review https://github.com/juju/juju/pull/15699

[Caner Derici (cderici)]

So with further investigation, we scratched off the #15699, as we realized that the way that zaza was doing this is incorrect, in that it uses the Charm field in the subordinate units to get the charm_url. This is an internal field and shouldn't be used externally. Furthermore the naming of this field is somewhat misleading, as it's really the "upgrading-from" field that's being set only during an upgrade -to indicate the previous charm url prior to the upgrade- and it is to be deleted/unset right after the upgrade is done. This is why getting the charm-url from this field is not correct.

Instead we put up the https://github.com/juju/python-libjuju/pull/879 on pylibjuju. It should make your job implementing "get_subordinate_units" on zaza much easier. @freyes it would help if you could confirm. Thanks!