set-temp-url-secret action fails with UnboundLocalError: local variable 'keystone_session' referenced before assignment

[Corey Bryant]

Przemyslaw Hausman reported:

When I run `set-temp-url-secret` action, it fails, and I get the following in the debug-log:

```
$ juju debug-log -i ironic-conductor/0
[...]
unit-ironic-conductor-0: 23:37:02 ERROR unit.ironic-conductor/0.juju-log action "set-temp-url-secret" failed: "local variable 'keystone_session' referenced before assignment" "Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ironic-conductor-0/charm/actions/set-temp-url-secret", line 112, in main
    action(args)
  File "/var/lib/juju/agents/unit-ironic-conductor-0/charm/actions/set-temp-url-secret", line 60, in set_temp_url_secret
    os_cli = api_utils.OSClients(keystone_session)
UnboundLocalError: local variable 'keystone_session' referenced before assignment

[Corey Bryant (corey.bryant)]

In a debug session with Natalia we were able to find a little more details:

Failed to create keystone session ("'PathDistribution' object has no attribute '_normalized_name'")

https://github.com/pypa/setuptools/issues/3452 which says to update to importlib_metadata 4.3 or later.

The version of importlib_metadata in the wheelhouse.txt is 2.1.3.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/881889

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/881889
Committed: https://opendev.org/openstack/charm-ironic-conductor/commit/9f221f1087c51f9323dbb41cba52018a93ccce5e
Submitter: "Zuul (22348)"
Branch: master

commit 9f221f1087c51f9323dbb41cba52018a93ccce5e
Author: Corey Bryant <email address hidden>
Date: Fri Apr 28 15:56:55 2023 -0400

    Fix exception in set-temp-url-secret action

    There are two issues fixed in this bug. The first is that the
    keystone_session exception block does not return on failure. The
    second, and the reason that create_keystone_session is failing,
    is due to the use of setuptools 60.9.0+ with the
    importlib-metadata in wheelhouse.txt that is pinned to <3.0.0.
    For more details see: https://github.com/pypa/setuptools/issues/3452

    Closes-Bug: #2018018
    Change-Id: I266c1fb92d531aded2f3253766de0a79accd9577

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/881666

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/881667

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/881666
Committed: https://opendev.org/openstack/charm-ironic-conductor/commit/6e3dbf2362a334adfacfb6a0dc3d94ab7dc38b6f
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 6e3dbf2362a334adfacfb6a0dc3d94ab7dc38b6f
Author: Corey Bryant <email address hidden>
Date: Fri Apr 28 15:56:55 2023 -0400

    Fix exception in set-temp-url-secret action

    There are two issues fixed in this bug. The first is that the
    keystone_session exception block does not return on failure. The
    second, and the reason that create_keystone_session is failing,
    is due to the use of setuptools 60.9.0+ with the
    importlib-metadata in wheelhouse.txt that is pinned to <3.0.0.
    For more details see: https://github.com/pypa/setuptools/issues/3452

    Closes-Bug: #2018018
    Change-Id: I266c1fb92d531aded2f3253766de0a79accd9577
    (cherry picked from commit 9f221f1087c51f9323dbb41cba52018a93ccce5e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/881667
Committed: https://opendev.org/openstack/charm-ironic-conductor/commit/8c2ed98bc1540cf97dc2b15e9e7499782acc1055
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 8c2ed98bc1540cf97dc2b15e9e7499782acc1055
Author: Corey Bryant <email address hidden>
Date: Fri Apr 28 15:56:55 2023 -0400

    Fix exception in set-temp-url-secret action

    There are two issues fixed in this bug. The first is that the
    keystone_session exception block does not return on failure. The
    second, and the reason that create_keystone_session is failing,
    is due to the use of setuptools 60.9.0+ with the
    importlib-metadata in wheelhouse.txt that is pinned to <3.0.0.
    For more details see: https://github.com/pypa/setuptools/issues/3452

    Closes-Bug: #2018018
    Change-Id: I266c1fb92d531aded2f3253766de0a79accd9577
    (cherry picked from commit 9f221f1087c51f9323dbb41cba52018a93ccce5e)

[Natalia Litvinova (natalytvinova)]

Unfortunately, the install hook fails for the charm:

root@juju-ac74ac-2-lxd-21:/var/lib/juju/agents/unit-ironic-conductor-5/charm# ./hooks/install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
build-essential is already the newest version (12.9ubuntu3).
python3-yaml is already the newest version (5.4.1-1ubuntu1).
python3-dev is already the newest version (3.10.6-1~22.04).
python3-setuptools is already the newest version (59.6.0-1.2ubuntu0.22.04.1).
python3-pip is already the newest version (22.0.2+dfsg-1ubuntu0.2).
python3-wheel is already the newest version (0.37.1-2ubuntu0.22.04.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
libffi-dev is already the newest version (3.4.2-4).
libpython3-dev is already the newest version (3.10.6-1~22.04).
libssl-dev is already the newest version (3.0.2-0ubuntu1.9).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Looking in links: wheelhouse
Requirement already satisfied: pip in /var/lib/juju/agents/unit-ironic-conductor-5/.venv/lib/python3.10/site-packages (22.0.4)
Looking in links: wheelhouse
Requirement already satisfied: setuptools in /var/lib/juju/agents/unit-ironic-conductor-5/.venv/lib/python3.10/site-packages (60.8.2)
Looking in links: wheelhouse
ERROR: Could not find a version that satisfies the requirement MarkupSafe==2.1.1 (from versions: none)
ERROR: No matching distribution found for MarkupSafe==2.1.1
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-ironic-conductor-5/charm/./hooks/install", line 8, in <module>
    basic.bootstrap_charm_deps()
  File "/var/lib/juju/agents/unit-ironic-conductor-5/charm/lib/charms/layer/basic.py", line 226, in bootstrap_charm_deps
    check_call([pip, 'install', '-U', reinstall_flag, '--no-index',
  File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/var/lib/juju/agents/unit-ironic-conductor-5/.venv/bin/pip', 'install', '-U', '--force-reinstall', '--no-index', '--no-cache-dir', '-f', 'wheelhouse', 'MarkupSafe==2.1.1', 'wheel==0.37.1']' returned non-zero exit status 1.
root@juju-ac74ac-2-lxd-21:/var/lib/juju/agents/unit-ironic-conductor-5/charm#

[Alex Kavanagh (ajkavanagh)]

@natalytvinova is this on the stable/zed ironic-conductor charm? There's another rebuild of the charm coming as it also has problems installing on s390x and ppc64le.

[Corey Bryant (corey.bryant)]

I gave this a deploy on amd64 and it is working for me. I wonder if (like Alex mentioned) this is a non-amd64 deploy. I have the ironic-conductor stable/zed rev 110 deployed.

[Corey Bryant (corey.bryant)]

Can you share your /var/lib/juju/agents/unit-ironic-conductor-*/charm/wheelhouse contents?

[Natalia Litvinova (natalytvinova)]

@corey.bryant @ajkavanagh this is stable/zed rev 111
This is the content:
ubuntu@juju-ac74ac-0-lxd-21:/var/lib/juju/agents/unit-ironic-conductor-3/charm/wheelhouse$ ls
attrs-22.1.0-py2.py3-none-any.whl os_service_types-1.7.0-py3-none-any.whl
certifi-2022.9.24-py3-none-any.whl packaging-21.3-py3-none-any.whl
cffi-1.15.1-cp310-cp310-linux_aarch64.whl pbr-5.10.0-py2.py3-none-any.whl
charmhelpers-1.1.1.dev47-py3-none-any.whl pip-22.0.4-py3-none-any.whl
charms.openstack-0.0.1.dev1-py3-none-any.whl prettytable-3.4.1-py3-none-any.whl
charms.reactive-1.5.1-py3-none-any.whl psutil-5.9.2-cp310-cp310-linux_aarch64.whl
charset_normalizer-2.1.1-py3-none-any.whl pyaml-21.10.1-py2.py3-none-any.whl
cryptography-38.0.1-cp310-cp310-linux_aarch64.whl pycparser-2.21-py2.py3-none-any.whl
Cython-0.29.32-cp310-cp310-linux_aarch64.whl pyOpenSSL-22.1.0-py3-none-any.whl
debtcollector-2.5.0-py3-none-any.whl pyparsing-3.0.9-py3-none-any.whl
dnspython-2.2.1-py3-none-any.whl pyrsistent-0.17.3-cp310-cp310-linux_aarch64.whl
flit_core-3.7.1-py3-none-any.whl python_glanceclient-3.6.0-py3-none-any.whl
idna-3.4-py3-none-any.whl python_keystoneclient-5.0.1-py3-none-any.whl
importlib_metadata-4.3.0-py3-none-any.whl python_swiftclient-4.1.0-py3-none-any.whl
importlib_resources-2.0.1-py2.py3-none-any.whl pytz-2022.4-py3-none-any.whl
iso8601-0.1.16-py3-none-any.whl PyYAML-5.3.1-cp310-cp310-linux_aarch64.whl
Jinja2-3.1.2-py3-none-any.whl requests-2.28.1-py3-none-any.whl
jsonpatch-1.32-py2.py3-none-any.whl rfc3986-2.0.0-py2.py3-none-any.whl
jsonpointer-2.3-py2.py3-none-any.whl setuptools-60.8.2-py3-none-any.whl
jsonschema-3.2.0-py2.py3-none-any.whl setuptools_scm-3.4.1-py2.py3-none-any.whl
keystoneauth1-5.0.0-py3-none-any.whl six-1.16.0-py2.py3-none-any.whl
MarkupSafe-2.1.1-cp310-cp310-linux_aarch64.whl stevedore-4.0.1-py3-none-any.whl
msgpack-1.0.4-cp310-cp310-linux_aarch64.whl toml-0.10.2-py2.py3-none-any.whl
netaddr-0.7.19-py3-none-any.whl urllib3-1.26.12-py2.py3-none-any.whl
netifaces-0.11.0-cp310-cp310-linux_aarch64.whl warlock-1.3.3-py2.py3-none-any.whl
oslo.config-9.0.0-py3-none-any.whl wcwidth-0.2.5-py2.py3-none-any.whl
oslo.i18n-5.1.0-py3-none-any.whl wheel-0.37.1-py2.py3-none-any.whl
oslo.serialization-5.0.0-py3-none-any.whl wrapt-1.14.1-cp310-cp310-linux_aarch64.whl
oslo.utils-6.0.1-py3-none-any.whl zipp-1.2.0-py2.py3-none-any.whl

[Corey Bryant (corey.bryant)]

With rev 111 I did hit the install issue. The only wheel available for MarkupSafe is for aarch64:

ubuntu@juju-d7dab6-test-0:~$ ls /var/lib/juju/agents/unit-ironic-conductor-0/charm/wheelhouse/MarkupSafe-2.1.1-cp310-cp310-linux_aarch64.whl
/var/lib/juju/agents/unit-ironic-conductor-0/charm/wheelhouse/MarkupSafe-2.1.1-cp310-cp310-linux_aarch64.whl

The reason for this is that with charms that use binary wheels, each architecture build produces a different revision. So the 111 revision is arm64 only.