Compute Component: Error: (pymysql.err.ProgrammingError) (1146, "Table 'nova_api.cell_mappings' doesn't exist")

[wes hayutin]

https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-compute-master/7dac4e0/logs/undercloud/var/log/extra/podman/containers/nova_db_sync/stdout.log.txt.gz

Is [api_database]/connection set in nova.conf?
Is the cell0 database connection URL correct?
Error: (pymysql.err.ProgrammingError) (1146, "Table 'nova_api.cell_mappings' doesn't exist")
[SQL: SELECT cell_mappings.created_at AS cell_mappings_created_at, cell_mappings.updated_at AS cell_mappings_updated_at, cell_mappings.id AS cell_mappings_id, cell_mappings.uuid AS cell_mappings_uuid, cell_mappings.name AS cell_mappings_name, cell_mappings.transport_url AS cell_mappings_transport_url, cell_mappings.database_connection AS cell_mappings_database_connection, cell_mappings.disabled AS cell_mappings_disabled
FROM cell_mappings
WHERE cell_mappings.uuid = %(uuid_1)s
 LIMIT %(param_1)s]
[parameters: {'uuid_1': '00000000-0000-0000-0000-000000000000', 'param_1': 1}]
(Background on this error at: http://sqlalche.me/e/14/f405)

https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-compute-master/7dac4e0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

+ echo 'Running command: '\''/usr/bin/bootstrap_host_exec nova_conductor su nova -s /bin/bash -c '\''/usr/bin/nova-manage db sync '\'''\'''
+ exec /usr/bin/bootstrap_host_exec nova_conductor su nova -s /bin/bash -c ''\''/usr/bin/nova-manage' db sync \'
2021-08-19 08:17:33.982762 | fa163e06-c6d2-5dfd-0459-00000000197e | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | standalone | error={"changed": false, "msg": "Failed containers: nova_api_db_sync, nova_api_map_cell0, nova_api_ensure_default_cell, nova_db_sync"}
2021-08-19 08:17:33.983320 | fa163e06-c6d2-5dfd-0459-00000000197e | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_3 | standalone | 0:19:23.159835 | 41.20s

[wes hayutin (weshayutin)]

hit the line on 8/19
http://dashboard-ci.tripleo.org/d/M1DYSidGz/component-pipeline?viewPanel=475&orgId=1

[melanie witt (melwitt)]

Here's an error that was raised during the sync of the nova_api database [1]:

Running command: '/usr/bin/bootstrap_host_exec nova_api su nova -s /bin/bash -c '/usr/bin/nova-manage api_db sync''
+ echo 'Running command: '\''/usr/bin/bootstrap_host_exec nova_api su nova -s /bin/bash -c '\''/usr/bin/nova-manage api_db sync'\'''\'''
+ exec /usr/bin/bootstrap_host_exec nova_api su nova -s /bin/bash -c ''\''/usr/bin/nova-manage' api_db 'sync'\'''
An error has occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/nova/cmd/manage.py", line 2845, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/usr/lib/python3.6/site-packages/nova/cmd/manage.py", line 584, in sync
    return migration.db_sync(version, database='api')
  File "/usr/lib/python3.6/site-packages/nova/db/migration.py", line 144, in db_sync
    config.set_main_option('sqlalchemy.url', str(engine.url))
  File "/usr/lib/python3.6/site-packages/alembic/config.py", line 243, in set_main_option
    self.set_section_option(self.config_ini_section, name, value)
  File "/usr/lib/python3.6/site-packages/alembic/config.py", line 270, in set_section_option
    self.file_config.set(section, name, value)
  File "/usr/lib64/python3.6/configparser.py", line 1193, in set
    super().set(section, option, value)
  File "/usr/lib64/python3.6/configparser.py", line 894, in set
    value)
  File "/usr/lib64/python3.6/configparser.py", line 402, in before_set
    "position %d" % (value, tmp_value.find('%')))
ValueError: invalid interpolation syntax in 'mysql+pymysql://nova_api:Jl6ip4yYTetmtrYWydWwJAcmI@192.168.24.3/nova_api?read_default_file=%2Fetc%2Fmy.cnf.d%2Ftripleo.cnf&read_default_group=tripleo' at position 91

Which would result in failure to create the nova_api database tables.

[1] https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-compute-master/7dac4e0/logs/undercloud/var/log/extra/podman/containers/nova_api_db_sync/stdout.log.txt.gz

[sean mooney (sean-k-mooney)]

this looks like a bug in nova which we think https://review.opendev.org/c/openstack/nova/+/805663 will resolve.
i have added nova to the bug and asigned it to me for now. stephen has already submitted ^ but ill work with them on it.

tl;dr we recently converted form sql alchamey migrate to alembic

as part fo that we have to override part of alembic default config and we intoduced a cast form

sqlalchemy.engine.url.URL to string

i believe that cast results in the / in being url encoded causing the error you are seeign in ooo

we dont see this in devstack since we dont set the read_defaut_file as far as i know in our jobs.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/805663
Committed: https://opendev.org/openstack/nova/commit/6c5dd864c24bc8cec89dccbff68b63c3bc23ef70
Submitter: "Zuul (22348)"
Branch: master

commit 6c5dd864c24bc8cec89dccbff68b63c3bc23ef70
Author: Stephen Finucane <email address hidden>
Date: Mon Aug 23 17:42:08 2021 +0100

    db: Handle parameters in DB strings

    As part of 'nova.db.migration', it is necessary to override the
    'sqlalchemy.url' config option used by alembic. We were doing this but
    we weren't handling the fact that we could receive encoded URL strings.
    This causes issues for the ConfigParser instead used under the hood, as
    noted in the alembic docs [1]:

      value – the value. Note that this value is passed to ConfigParser.set,
      which supports variable interpolation using pyformat (e.g.
      %(some_value)s). A raw percent sign not part of an interpolation
      symbol must therefore be escaped, e.g. %%. The given value may refer
      to another value already in the file using the interpolation format.

    Resolve the issue by escaping % with %%. The engine.url is always
    encoded therefore this can be done unconditionally.

    [1] https://alembic.sqlalchemy.org/en/latest/api/config.html#alembic.config.Config.set_main_option

    Closes-Bug: #1940555
    Change-Id: I74de55107a80af13df348f2bce49415b08028746
    Signed-off-by: Stephen Finucane <email address hidden>
    Co-Authored-By: Sean Mooney <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 24.0.0.0rc1 release candidate.

[Ronelle Landy (rlandy)]

no trace in skiplist - closing out the tripleo branch