[3.0-rc2] glance-simplestreams-sync status unknown

[Bas de Bruijne]

Testrun https://solutions.qa.canonical.com/testruns/testRun/a98057c3-a860-4096-b592-a474462cafbd (openstack yoga focal with juju 3.0rc2) fails because glance-simplestreams-sync is in unknown state:

```
2022-10-19-17:30:27 root ERROR [localhost] Command failed: juju wait-for unit -m foundations-maas:openstack --timeout 3600s glance-simplestreams-sync/0 --query 'workload-status=="active"'
2022-10-19-17:30:27 root ERROR [localhost] STDOUT follows:
properties:
  workload-status: unknown

2022-10-19-17:30:27 root ERROR [localhost] STDERR follows:
Unit "glance-simplestreams-sync/0" is running
ERROR timed out waiting for "glance-simplestreams-sync/0" to reach goal state
```

In the unit logs we see:
```
2022-10-19 17:34:23 INFO juju.worker.uniter.operation runhook.go:176 skipped "update-status" hook (missing)
2022-10-19 17:34:23 DEBUG juju.worker.uniter.operation executor.go:135 committing operation "run update-status hook" for glance-simplestreams-sync/0
2022-10-19 17:34:23 DEBUG juju.worker.uniter.operation executor.go:124 lock released for glance-simplestreams-sync/0
```

All the OpenStack testrun have run into this problem so far.

Logs and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/a98057c3-a860-4096-b592-a474462cafbd/index.html

[Alex Kavanagh (ajkavanagh)]

Hi Bas, is this pre- or post- vault initialisation? Pre certificates, the gss charms does indeed stay in the "unknown" status.

[Bas de Bruijne (basdbruijne)]

Vault is already initialized at this point. We have seen this work many times on juju 2.9 so I suspect there is some communication issue with juju 3.

[Bas de Bruijne (basdbruijne)]

To give some more context, FCE initializes vault, ceilometer, octavia, heat, cinder, designate, and keystone and waits for the deployment to idle in between each step. Then we run:

```
2022-10-19-16:30:19 fce.openstack INFO Starting step: openstack:glance_simplestreams_sync
2022-10-19-16:30:21 root DEBUG [localhost]: juju config -m foundations-maas:openstack glance-simplestreams-sync run=true
2022-10-19-16:30:22 root DEBUG [localhost]: juju status -m foundations-maas:openstack --format=yaml
2022-10-19-16:30:27 root DEBUG [localhost]: juju wait-for unit -m foundations-maas:openstack --timeout 3600s glance-simplestreams-sync/0 --query 'workload-status=="active"'
2022-10-19-17:30:27 root ERROR [localhost] Command failed: juju wait-for unit -m foundations-maas:openstack --timeout 3600s glance-simplestreams-sync/0 --query 'workload-status=="active"'
2022-10-19-17:30:27 root ERROR [localhost] STDOUT follows:
properties:
  workload-status: unknown

2022-10-19-17:30:27 root ERROR [localhost] STDERR follows:
Unit "glance-simplestreams-sync/0" is running
ERROR timed out waiting for "glance-simplestreams-sync/0" to reach goal state
```

Usually gss syncs all the images within an hour but on all the juju 3.0 runs it doesn't change its status.

[Alex Kavanagh (ajkavanagh)]

So the issue is:

PythonArgs: ['/usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py']
Traceback:
 Traceback (most recent call last):
   File "/usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py", line 633, in <module>
     sys.exit(main())
   File "/usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py", line 599, in main
     update_product_streams_service(ksc, services, region_name)
   File "/usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py", line 383, in update_product_streams_service
     update_endpoint_urls(
   File "/usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py", line 498, in update_endpoint_urls
     for rid in juju_run_cmd(['relation-ids', 'identity-service']).split():
   File "/usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py", line 472, in juju_run_cmd
     out = subprocess.check_output(_cmd)
   File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
   File "/usr/lib/python3.8/subprocess.py", line 493, in run
     with Popen(*popenargs, **kwargs) as process:
   File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
     self._execute_child(args, executable, preexec_fn, close_fds,
   File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
     raise child_exception_type(errno_num, err_msg, err_filename)
 FileNotFoundError: [Errno 2] No such file or directory: 'juju-run'

I notice that this is running on juju 3.0rc which I guess isn't supported yet.

[Alex Kavanagh (ajkavanagh)]

I've set it to wishlist as it'll need to be a new change to actually make it compatible with juju 3.0.; it may be necessary to change this to "high" once Juju 3.0 becomes the default install.

[Ian Booth (wallyworld)]

@alex, in juju 3.0, the command is called juju-exec

[Alex Kavanagh (ajkavanagh)]

@Ian, thanks; is there a migration doc that explains new-for-old commands?

[Billy Olsen (billy-olsen)]

@Alex - https://juju.is/docs/olm/roadmap#heading--juju-3-0-0---22-oct-2022

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/866493
Committed: https://opendev.org/openstack/charm-glance-simplestreams-sync/commit/9304d1543463ce65f1e2fae9cba1cea22184e3c0
Submitter: "Zuul (22348)"
Branch: master

commit 9304d1543463ce65f1e2fae9cba1cea22184e3c0
Author: Bas de Bruijne <email address hidden>
Date: Fri Dec 2 13:30:41 2022 -0400

    Add juju-exec for juju 3 compatibility

    In juju3, juju-run is moved to juju-exec
    gsss should be able to handle both

    Closes-bug: #1993652
    Change-Id: I980306269a68c04735811324f6b826622bc6647c

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/867253

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/867254

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/867253
Committed: https://opendev.org/openstack/charm-glance-simplestreams-sync/commit/a71f14b8e978927ab9285b49d70a32f730bb5fa4
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit a71f14b8e978927ab9285b49d70a32f730bb5fa4
Author: Bas de Bruijne <email address hidden>
Date: Fri Dec 2 13:30:41 2022 -0400

    Add juju-exec for juju 3 compatibility

    In juju3, juju-run is moved to juju-exec
    gsss should be able to handle both

    Closes-bug: #1993652
    Change-Id: I980306269a68c04735811324f6b826622bc6647c
    (cherry picked from commit 9304d1543463ce65f1e2fae9cba1cea22184e3c0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/867254
Committed: https://opendev.org/openstack/charm-glance-simplestreams-sync/commit/8aac05d48700604ebeefd5eecdcf40ba1b444a9c
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 8aac05d48700604ebeefd5eecdcf40ba1b444a9c
Author: Bas de Bruijne <email address hidden>
Date: Fri Dec 2 13:30:41 2022 -0400

    Add juju-exec for juju 3 compatibility

    In juju3, juju-run is moved to juju-exec
    gsss should be able to handle both

    Closes-bug: #1993652
    Change-Id: I980306269a68c04735811324f6b826622bc6647c
    (cherry picked from commit 9304d1543463ce65f1e2fae9cba1cea22184e3c0)