Comment 1 for bug 2042838

in this run, https://solutions.qa.canonical.com/testruns/2a888825-e0e9-4230-9704-9e1372ffb4a1

keystone is blocked because "(workload) Error in charm (see logs): Bootstrap failed"

debug log
unit-keystone-1: 11:18:47 ERROR unit.keystone/1.juju-log Exited with code 1. Stderr:
unit-keystone-1: 11:18:47 ERROR unit.keystone/1.juju-log Error occurred bootstrapping keystone service
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-keystone-1/charm/src/utils/manager.py", line 326, in _bootstrap
    self.run_cmd(
  File "/var/lib/juju/agents/unit-keystone-1/charm/src/utils/manager.py", line 65, in run_cmd
    return pebble_handler.execute(cmd, exception_on_error=True, **kwargs)
  File "/var/lib/juju/agents/unit-keystone-1/charm/venv/ops_sunbeam/container_handlers.py", line 206, in execute
    stdout, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-keystone-1/charm/venv/ops/pebble.py", line 1374, in wait_output
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['keystone-manage', 'bootstrap', '--bootstrap-username', '_charm-keystone-admin', '--bootstrap-password', 'IbKvMPoMrMuN', '--bootstrap-project-name', 'admin', '--bootstrap-role-name', 'Admin', '--bootstrap-service-name', 'keystone', '--bootstrap-admin-url', 'http://10.152.183.91:5000/v3', '--bootstrap-public-url', 'http://10.246.167.217/openstack-keystone/v3', '--bootstrap-internal-url', 'http://10.246.167.218/openstack-keystone/v3', '--bootstrap-region-id', 'RegionOne'], stdout='(oslo_db.sqlalchemy.engines): 2023-11-26 11:16:57,099 WARNING SQL connection failed. 10 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:07,114 WARNING SQL connection failed. 9 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:17,124 WARNING SQL connection failed. 8 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:27,134 WARNING SQL connection failed. 7 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:37,145 WARNING SQL connection failed. 6 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:47,157 WARNING SQL connection failed. 5 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:57,169 WARNING SQL connection failed. 4 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:18:07,181 WARNING SQL connection failed. 3 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:18:17,193 WARNING SQL connection failed. 2 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:18:37,210 WARNING SQL connection failed. 1 attempts left.\n(ke' [truncated], stderr=''
unit-keystone-1: 11:18:47 ERROR unit.keystone/1.juju-log Exception raised in section 'Bootstrapping': Bootstrap failed
unit-placement-mysql-router-1: 11:18:47 INFO juju.worker.uniter awaiting error resolution for "start" hook
unit-keystone-1: 11:18:47 ERROR unit.keystone/1.juju-log Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-keystone-1/charm/src/utils/manager.py", line 326, in _bootstrap
    self.run_cmd(
  File "/var/lib/juju/agents/unit-keystone-1/charm/src/utils/manager.py", line 65, in run_cmd
    return pebble_handler.execute(cmd, exception_on_error=True, **kwargs)
  File "/var/lib/juju/agents/unit-keystone-1/charm/venv/ops_sunbeam/container_handlers.py", line 206, in execute
    stdout, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-keystone-1/charm/venv/ops/pebble.py", line 1374, in wait_output
    raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['keystone-manage', 'bootstrap', '--bootstrap-username', '_charm-keystone-admin', '--bootstrap-password', 'IbKvMPoMrMuN', '--bootstrap-project-name', 'admin', '--bootstrap-role-name', 'Admin', '--bootstrap-service-name', 'keystone', '--bootstrap-admin-url', 'http://10.152.183.91:5000/v3', '--bootstrap-public-url', 'http://10.246.167.217/openstack-keystone/v3', '--bootstrap-internal-url', 'http://10.246.167.218/openstack-keystone/v3', '--bootstrap-region-id', 'RegionOne'], stdout='(oslo_db.sqlalchemy.engines): 2023-11-26 11:16:57,099 WARNING SQL connection failed. 10 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:07,114 WARNING SQL connection failed. 9 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:17,124 WARNING SQL connection failed. 8 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:27,134 WARNING SQL connection failed. 7 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:37,145 WARNING SQL connection failed. 6 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:47,157 WARNING SQL connection failed. 5 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:17:57,169 WARNING SQL connection failed. 4 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:18:07,181 WARNING SQL connection failed. 3 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:18:17,193 WARNING SQL connection failed. 2 attempts left.\n(oslo_db.sqlalchemy.engines): 2023-11-26 11:18:37,210 WARNING SQL connection failed. 1 attempts left.\n(ke' [truncated], stderr=''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-keystone-1/charm/venv/ops_sunbeam/guard.py", line 91, in guard
    yield
  File "/var/lib/juju/agents/unit-keystone-1/charm/venv/ops_sunbeam/charm.py", line 292, in configure_charm
    self.configure_app(event)
  File "/var/lib/juju/agents/unit-keystone-1/charm/venv/ops_sunbeam/charm.py", line 274, in configure_app
    self.configure_app_leader(event)
  File "/var/lib/juju/agents/unit-keystone-1/charm/./src/charm.py", line 1424, in configure_app_leader
    self.keystone_bootstrap()
  File "/var/lib/juju/agents/unit-keystone-1/charm/venv/ops_sunbeam/job_ctrl.py", line 74, in wrapped_f
    f(charm, *args, **kwargs)
  File "/var/lib/juju/agents/unit-keystone-1/charm/./src/charm.py", line 1395, in keystone_bootstrap
    self.keystone_manager.setup_keystone()
  File "/var/lib/juju/agents/unit-keystone-1/charm/src/utils/manager.py", line 134, in setup_keystone
    self._bootstrap()
  File "/var/lib/juju/agents/unit-keystone-1/charm/src/utils/manager.py", line 352, in _bootstrap
    raise KeystoneExceptionError("Bootstrap failed")
utils.client.KeystoneExceptionError: Bootstrap failed