ironic-conductor stays blocked with "invalid enabled-deploy-interfaces config"

[Bas de Bruijne]

Testrun https://solutions.qa.canonical.com/v2/testruns/2cf79fc7-cd11-41d8-9b1b-5eefa9bfbaf0 fails because ironic-conductor stays blocked with:

```
ironic-conductor/0* blocked idle 4/lxd/6 10.246.165.83 invalid enabled-deploy-interfaces config, run set-temp-url-secret action on leader to enable direct deploy method
  filebeat/78 active idle 10.246.165.83 Filebeat ready.
  ironic-conductor-mysql-router/0* active idle 10.246.165.83 Unit is ready
  landscape-client/78 maintenance idle 10.246.165.83 Need computer-title and juju-info to proceed
  nrpe/86 active idle 10.246.165.83 icmp,5666/tcp Ready
  prometheus-grok-exporter/78 active idle 10.246.165.83 9144/tcp Unit is ready
  telegraf/78 active idle 10.246.165.83 9103/tcp Monitoring ironic-conductor/0 (source version/commit d208a64)
```

enabled-deploy-interfaces is not set in the bundle, not does it show up in `juju export-bundle` (see the artifacts [1] for both), so this message is a little weird.

In the crashdump we see no mention of enabled-deploy-interfaces in particular, but we do see some SQL errors:
```
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last):
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines connection.scalar(select(1))
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1171, in scalar
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines return self.execute(object_, *multiparams, **params).scalar()
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1289, in execute
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines ret = self._execute_context(
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines self._handle_dbapi_exception(
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2024, in _handle_dbapi_exception
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines util.raise_(newraise, with_traceback=exc_info[2], from_=e)
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 207, in raise_
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines raise exception
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines self.dialect.do_execute(
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters)
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 148, in execute
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines result = self._query(query)
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/cursors.py", line 310, in _query
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines conn.query(q)
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 548, in query
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 775, in _read_query_result
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines result.read()
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 1156, in read
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet()
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 692, in _read_packet
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines packet_header = self._read_bytes(4)
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 748, in _read_bytes
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines raise err.OperationalError(
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines [SQL: SELECT 1]
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines (Background on this error at: https://sqlalche.me/e/14/e3q8)
2022-12-13 02:14:27.657 75119 ERROR oslo_db.sqlalchemy.engines
2022-12-13 02:15:06.714 75119 WARNING amqp [-] Received method (60, 30) during closing channel 1. This method will be ignored
2022-12-13 02:15:06.742 75119 INFO ironic.conductor.base_manager [-] Successfully stopped conductor with hostname juju-77dca4-4-lxd-6.nosilo.lab1.solutionsqa.
2022-12-13 02:15:06.742 75119 INFO ironic.conductor.base_manager [-] Successfully shut down periodic tasks
2022-12-13 02:15:06.742 75119 INFO ironic.common.rpc_service [-] Stopped RPC server for service ironic.conductor_manager on host juju-77dca4-4-lxd-6.nosilo.lab1.solutionsqa.
2022-12-13 02:15:08.601 100461 WARNING oslo_config.cfg [-] Deprecated: Option "sql_connection" from group "database" is deprecated. Use option "connection" from group "database"
```

This testrun is yoga/stable on jammy.

[1] Crashdumps and configs: https://oil-jenkins.canonical.com/artifacts/2cf79fc7-cd11-41d8-9b1b-5eefa9bfbaf0/index.html

[Corey Bryant (corey.bryant)]

Hi Bas,

I agree, the message could use an update to be more clear. I'll triage this bug with the expectation being that the message could use better clarity. Please let us know if you were looking for any other updates.

Thanks,
Corey

[Jeffrey Chang (modern911)]

Here is another recent run, https://solutions.qa.canonical.com/testruns/605c16f6-dd69-403e-bca8-b47465923fdb
ironic-conductor is blocked with the same error, but debug log is different

2024-03-26 02:03:45 DEBUG juju.worker.uniter.operation executor.go:135 executing operation "run action 38" for ironic-conductor/0
2024-03-26 02:03:45 DEBUG juju.worker.uniter agent.go:22 [AGENT-STATUS] executing: running action set-temp-url-secret
2024-03-26 02:03:45 DEBUG juju.worker.uniter.runner runner.go:378 running action "set-temp-url-secret" on 1
2024-03-26 02:03:45 DEBUG juju.worker.uniter.runner runner.go:719 starting jujuc server {unix @/var/lib/juju/agents/unit-ironic-conductor-0/agent.socket <nil>}
2024-03-26 02:03:46 ERROR unit.ironic-conductor/0.juju-log server.go:325 action "set-temp-url-secret" failed: "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: undefined symbol: FIPS_mode" "Traceback (mos
t recent call last):
  File "/var/lib/juju/agents/unit-ironic-conductor-0/charm/actions/set-temp-url-secret", line 112, in main
    action(args)
  File "/var/lib/juju/agents/unit-ironic-conductor-0/charm/actions/set-temp-url-secret", line 60, in set_temp_url_secret
    os_cli = api_utils.OSClients(keystone_session)
  File "/var/lib/juju/agents/unit-ironic-conductor-0/charm/lib/charm/openstack/ironic/api_utils.py", line 53, in __init__
    self._img_cli = glanceclient.Client(
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/glanceclient/client.py", line 61, in Client
    module = importutils.import_versioned_module('glanceclient', int(version),
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/oslo_utils/importutils.py", line 97, in import_versioned_module
    return import_module(module_str)
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/oslo_utils/importutils.py", line 73, in import_module
    __import__(import_str)
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/glanceclient/v2/__init__.py", line 15, in <module>
    from glanceclient.v2.client import Client # noqa
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/glanceclient/v2/client.py", line 17, in <module>
    from glanceclient.common import http
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/glanceclient/common/http.py", line 23, in <module>
    import OpenSSL
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/OpenSSL/crypto.py", line 17, in <module>
    from OpenSSL._util import (
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/cryptography/hazmat/binding...

[Alex Kavanagh (ajkavanagh)]

This bit:

  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/glanceclient/common/http.py", line 23, in <module>
    import OpenSSL
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/OpenSSL/crypto.py", line 17, in <module>
    from OpenSSL._util import (
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 14, in <module>
    from cryptography.hazmat.bindings._openssl import ffi, lib
ImportError: /var/lib/juju/agents/unit-ironic-conductor-0/.venv/lib/python3.10/site-packages/cryptography/hazmat/bindings/_openssl.abi3.so: undefined symbol: FIPS_mode

(i.e. undefined symbol FIPS_mode)

looks like a build issue. I think keystone-saml-mellon had/has a similar issue: https://bugs.launchpad.net/charm-keystone-saml-mellon/+bug/1975491 which also lists ironic-conductor as fixed. I wonder if something has changed (again)?

[Jeffrey Chang (modern911)]

SQA still saw the FIPS related error with ironic-conductor rev 173 ed9b7ef, several times since late March.
On of the Testrun - https://solutions.qa.canonical.com/testruns/823bc8b9-9d02-4564-bc6e-a6d6390e17ce