First time I've seen this, logging to track in case we see it again. Full artifact collection: https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_smoke/openstack/charm-octavia/759185/9/19832/index.html
Looking at the juju status at the time it appears to me that the database units have just completed clustering, so perhaps this is a race or something firing too early?
2020-10-28 22:45:47 DEBUG juju-log shared-db:18: 0 section(s) found
2020-10-28 22:45:47 INFO juju-log shared-db:18: Registered config file: /etc/nova/nova.conf
2020-10-28 22:45:47 INFO juju-log shared-db:18: Registered config file: /etc/nova/api-paste.ini
2020-10-28 22:45:47 INFO juju-log shared-db:18: Registered config file: /etc/nova/vendor_data.json
2020-10-28 22:45:47 INFO juju-log shared-db:18: Registered config file: /etc/haproxy/haproxy.cfg
2020-10-28 22:45:47 INFO juju-log shared-db:18: Registered config file: /etc/apache2/sites-available/openstack_https_frontend.conf
2020-10-28 22:45:47 INFO juju-log shared-db:18: Registered config file: /etc/apache2/ports.conf
2020-10-28 22:45:47 INFO juju-log shared-db:18: Registered config file: /etc/memcached.conf
2020-10-28 22:45:47 INFO juju-log shared-db:18: Registered config file: /etc/apache2/sites-enabled/wsgi-api-os-compute.conf
2020-10-28 22:45:47 INFO juju-log shared-db:18: Registered config file: /etc/apache2/sites-enabled/wsgi-openstack-metadata.conf
2020-10-28 22:45:51 DEBUG juju-log shared-db:18: 0 section(s) found
2020-10-28 22:45:51 DEBUG juju-log shared-db:18: Generating template context for amqp
2020-10-28 22:45:52 DEBUG juju-log shared-db:18: Generating template context for image-service.
2020-10-28 22:45:52 DEBUG juju-log shared-db:18: 0 section(s) found
2020-10-28 22:45:52 INFO juju-log shared-db:18: HAProxy context is incomplete, this unit has no peers.
2020-10-28 22:45:53 DEBUG juju-log shared-db:18: Generating template context for identity-service
2020-10-28 22:45:53 INFO juju-log shared-db:18: Missing required data: service_port service_host auth_host auth_port admin_tenant_name admin_user admin_password
2020-10-28 22:45:53 DEBUG shared-db-relation-changed none
2020-10-28 22:45:54 INFO juju-log shared-db:18: Generating template context from neutron api relation
2020-10-28 22:45:54 DEBUG juju-log shared-db:18: Generating template context for identity-service
2020-10-28 22:45:55 INFO juju-log shared-db:18: Missing required data: service_port service_host auth_host auth_port admin_tenant_name admin_user admin_password
2020-10-28 22:45:55 DEBUG juju-log shared-db:18: Ensuring haproxy enabled in /etc/default/haproxy.
2020-10-28 22:45:56 INFO juju-log shared-db:18: HAProxy context is incomplete, this unit has no peers.
2020-10-28 22:45:56 DEBUG juju-log shared-db:18: 0 section(s) found
2020-10-28 22:45:56 DEBUG shared-db-relation-changed none
2020-10-28 22:45:56 INFO juju-log shared-db:18: HAProxy context is incomplete, this unit has no peers.
2020-10-28 22:45:56 DEBUG shared-db-relation-changed none
2020-10-28 22:45:57 INFO juju-log shared-db:18: HAProxy context is incomplete, this unit has no peers.
2020-10-28 22:45:57 INFO juju-log shared-db:18: Generating template context from neutron api relation
2020-10-28 22:45:57 DEBUG juju-log shared-db:18: Checking for maintenance notifications
2020-10-28 22:45:57 DEBUG juju-log shared-db:18: Generating template context for amqp
2020-10-28 22:45:57 DEBUG juju-log shared-db:18: Generating template context for image-service.
2020-10-28 22:45:57 DEBUG juju-log shared-db:18: 0 section(s) found
2020-10-28 22:45:57 INFO juju-log shared-db:18: HAProxy context is incomplete, this unit has no peers.
2020-10-28 22:45:58 DEBUG juju-log shared-db:18: Generating template context for identity-service
2020-10-28 22:45:58 INFO juju-log shared-db:18: Missing required data: service_port service_host auth_host auth_port admin_tenant_name admin_user admin_password
2020-10-28 22:45:58 DEBUG shared-db-relation-changed none
2020-10-28 22:45:58 INFO juju-log shared-db:18: Generating template context from neutron api relation
2020-10-28 22:45:59 INFO juju-log shared-db:18: Loaded template from templates/train/nova.conf
2020-10-28 22:45:59 INFO juju-log shared-db:18: Rendering from template: /etc/nova/nova.conf
2020-10-28 22:45:59 INFO juju-log shared-db:18: Wrote template /etc/nova/nova.conf.
2020-10-28 22:45:59 DEBUG juju-log shared-db:18: Generating template context for identity-service
2020-10-28 22:46:00 INFO juju-log shared-db:18: Missing required data: service_port service_host auth_host auth_port admin_tenant_name admin_user admin_password
2020-10-28 22:46:00 INFO juju-log shared-db:18: Loaded template from templates/mitaka/api-paste.ini
2020-10-28 22:46:00 INFO juju-log shared-db:18: Rendering from template: /etc/nova/api-paste.ini
2020-10-28 22:46:00 INFO juju-log shared-db:18: Wrote template /etc/nova/api-paste.ini.
2020-10-28 22:46:00 INFO juju-log shared-db:18: Loaded template from /var/lib/juju/agents/unit-nova-cloud-controller-0/charm/charmhelpers/contrib/openstack/templates/vendor_data.json
2020-10-28 22:46:00 INFO juju-log shared-db:18: Rendering from template: /etc/nova/vendor_data.json
2020-10-28 22:46:00 INFO juju-log shared-db:18: Wrote template /etc/nova/vendor_data.json.
2020-10-28 22:46:01 DEBUG juju-log shared-db:18: Ensuring haproxy enabled in /etc/default/haproxy.
2020-10-28 22:46:01 INFO juju-log shared-db:18: HAProxy context is incomplete, this unit has no peers.
2020-10-28 22:46:01 INFO juju-log shared-db:18: Loaded template from /var/lib/juju/agents/unit-nova-cloud-controller-0/charm/charmhelpers/contrib/openstack/templates/haproxy.cfg
2020-10-28 22:46:01 INFO juju-log shared-db:18: Rendering from template: /etc/haproxy/haproxy.cfg
2020-10-28 22:46:02 INFO juju-log shared-db:18: Wrote template /etc/haproxy/haproxy.cfg.
2020-10-28 22:46:02 INFO juju-log shared-db:18: Loaded template from /var/lib/juju/agents/unit-nova-cloud-controller-0/charm/charmhelpers/contrib/openstack/templates/openstack_https_frontend.conf
2020-10-28 22:46:02 INFO juju-log shared-db:18: Rendering from template: /etc/apache2/sites-available/openstack_https_frontend.conf
2020-10-28 22:46:02 INFO juju-log shared-db:18: Wrote template /etc/apache2/sites-available/openstack_https_frontend.conf.
2020-10-28 22:46:02 INFO juju-log shared-db:18: Loaded template from templates/ports.conf
2020-10-28 22:46:02 INFO juju-log shared-db:18: Rendering from template: /etc/apache2/ports.conf
2020-10-28 22:46:02 INFO juju-log shared-db:18: Wrote template /etc/apache2/ports.conf.
2020-10-28 22:46:02 INFO juju-log shared-db:18: Loaded template from /var/lib/juju/agents/unit-nova-cloud-controller-0/charm/charmhelpers/contrib/openstack/templates/memcached.conf
2020-10-28 22:46:02 INFO juju-log shared-db:18: Rendering from template: /etc/memcached.conf
2020-10-28 22:46:02 INFO juju-log shared-db:18: Wrote template /etc/memcached.conf.
2020-10-28 22:46:02 DEBUG shared-db-relation-changed none
2020-10-28 22:46:03 INFO juju-log shared-db:18: HAProxy context is incomplete, this unit has no peers.
2020-10-28 22:46:03 INFO juju-log shared-db:18: Loaded template from templates/wsgi-api-os-compute.conf
2020-10-28 22:46:03 INFO juju-log shared-db:18: Rendering from template: /etc/apache2/sites-enabled/wsgi-api-os-compute.conf
2020-10-28 22:46:03 INFO juju-log shared-db:18: Wrote template /etc/apache2/sites-enabled/wsgi-api-os-compute.conf.
2020-10-28 22:46:03 DEBUG shared-db-relation-changed none
2020-10-28 22:46:03 INFO juju-log shared-db:18: HAProxy context is incomplete, this unit has no peers.
2020-10-28 22:46:04 INFO juju-log shared-db:18: Generating template context from neutron api relation
2020-10-28 22:46:04 INFO juju-log shared-db:18: Loaded template from /var/lib/juju/agents/unit-nova-cloud-controller-0/charm/charmhelpers/contrib/openstack/templates/wsgi-openstack-metadata.conf
2020-10-28 22:46:04 INFO juju-log shared-db:18: Rendering from template: /etc/apache2/sites-enabled/wsgi-openstack-metadata.conf
2020-10-28 22:46:04 INFO juju-log shared-db:18: Wrote template /etc/apache2/sites-enabled/wsgi-openstack-metadata.conf.
2020-10-28 22:46:04 DEBUG juju-log shared-db:18: Database is NOT initialised
2020-10-28 22:46:04 DEBUG juju-log shared-db:18: Generating template context for amqp
2020-10-28 22:46:05 INFO juju-log shared-db:18: Generating template context for cell v2 share-db
2020-10-28 22:46:05 INFO juju-log shared-db:18: Migrating the nova-api database.
2020-10-28 22:46:06 DEBUG shared-db-relation-changed Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
2020-10-28 22:46:16 DEBUG shared-db-relation-changed /usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (3719, "'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.")
2020-10-28 22:46:16 DEBUG shared-db-relation-changed result = self._query(query)
2020-10-28 22:46:16 DEBUG shared-db-relation-changed /usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (1681, 'Integer display width is deprecated and will be removed in a future release.')
2020-10-28 22:46:16 DEBUG shared-db-relation-changed result = self._query(query)
2020-10-28 22:46:16 DEBUG shared-db-relation-changed /usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (3778, "'utf8_bin' is a collation of the deprecated character set UTF8MB3. Please consider using UTF8MB4 with an appropriate collation instead.")
2020-10-28 22:46:16 DEBUG shared-db-relation-changed result = self._query(query)
2020-10-28 22:46:17 INFO juju-log shared-db:18: Creating cell0 database records
2020-10-28 22:46:19 DEBUG shared-db-relation-changed Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
2020-10-28 22:46:19 DEBUG shared-db-relation-changed /usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (3719, "'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.")
2020-10-28 22:46:19 DEBUG shared-db-relation-changed result = self._query(query)
2020-10-28 22:46:20 INFO juju-log shared-db:18: Creating cell1 database records
2020-10-28 22:46:21 DEBUG shared-db-relation-changed Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
2020-10-28 22:46:22 DEBUG shared-db-relation-changed /usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (3719, "'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.")
2020-10-28 22:46:22 DEBUG shared-db-relation-changed result = self._query(query)
2020-10-28 22:46:22 INFO juju-log shared-db:18: cell1 was successfully created
2020-10-28 22:46:22 INFO juju-log shared-db:18: Migrating the nova database.
2020-10-28 22:46:24 DEBUG shared-db-relation-changed Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
2020-10-28 22:47:38 DEBUG shared-db-relation-changed /usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (3719, "'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.")
2020-10-28 22:47:38 DEBUG shared-db-relation-changed result = self._query(query)
2020-10-28 22:47:38 DEBUG shared-db-relation-changed /usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (3719, "'utf8' is currently an alias for the character set UTF8MB3, but will be an alias for UTF8MB4 in a future release. Please consider using UTF8MB4 in order to be unambiguous.")
2020-10-28 22:47:38 DEBUG shared-db-relation-changed result = self._query(query)
2020-10-28 22:47:38 DEBUG shared-db-relation-changed /usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (1831, "Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'nova_cell0.block_device_mapping'. This is deprecated and will be disallowed in a future release.")
2020-10-28 22:47:38 DEBUG shared-db-relation-changed result = self._query(query)
2020-10-28 22:47:38 DEBUG shared-db-relation-changed /usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (1681, 'Integer display width is deprecated and will be removed in a future release.')
2020-10-28 22:47:38 DEBUG shared-db-relation-changed result = self._query(query)
2020-10-28 22:47:38 DEBUG shared-db-relation-changed /usr/lib/python3/dist-packages/pymysql/cursors.py:170: Warning: (1831, "Duplicate index 'uniq_instances0uuid' defined on the table 'nova_cell0.instances'. This is deprecated and will be disallowed in a future release.")
2020-10-28 22:47:38 DEBUG shared-db-relation-changed result = self._query(query)
2020-10-28 22:47:38 ERROR juju-log shared-db:18: db sync failed
b'ERROR: Could not access cell0.\nHas the nova_api database been created?\nHas the nova_cell0 database been created?\nHas "nova-manage api_db sync" been run?\nHas "nova-manage cell_v2 map_cell0" been run?\nIs [api_database]/connection set in nova.conf?\nIs the cell0 database connection URL correct?\nError: (pymysql.err.OperationalError) (2013, \'Lost connection to MySQL server during query\')\n[SQL: \nALTER TABLE shadow_pci_devices ADD numa_node INTEGER]\n(Background on this error at: http://sqlalche.me/e/e3q8)\n'
2020-10-28 22:47:38 INFO juju-log shared-db:18: Retrying 'migrate_nova_databases' 5 more times (delay=3)
2020-10-28 22:47:41 DEBUG juju-log shared-db:18: Generating template context for amqp
2020-10-28 22:47:42 INFO juju-log shared-db:18: Generating template context for cell v2 share-db
2020-10-28 22:47:42 INFO juju-log shared-db:18: Migrating the nova-api database.
2020-10-28 22:47:43 DEBUG shared-db-relation-changed Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
2020-10-28 22:47:44 INFO juju-log shared-db:18: Ignoring CalledProcessError during nova-api database migration
b'An error has occurred:\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect\n return fn()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 303, in unique_connection\n return _ConnectionFairy._checkout(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 760, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 492, in checkout\n rec = pool._do_get()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 139, in _do_get\n self._dec_overflow()\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 136, in _do_get\n return self._create_connection()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 308, in _create_connection\n return _ConnectionRecord(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 437, in __init__\n self.__connect(first_connect_check=True)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 639, in __connect\n connection = pool._invoke_creator(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect\n return dialect.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 482, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 94, in Connect\n return Connection(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 325, in __init__\n self.connect()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 598, in connect\n self._get_server_information()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 975, in _get_server_information\n packet = self._read_packet()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet\n packet.check_error()\n File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error\n err.raise_mysql_exception(self._data)\n File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception\n raise errorclass(errno, errval)\npymysql.err.InternalError: (2003, "Can\'t connect to remote MySQL server for client connected to \'127.0.0.1:3306\'")\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/migration.py", line 93, in db_version\n return _db_version(repository, database, context)\n File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/migration.py", line 100, in _db_version\n return versioning_api.db_version(get_engine(database, context=context),\n File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/migration.py", line 43, in get_engine\n return db_session.get_api_engine()\n File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 151, in get_api_engine\n return api_context_manager.writer.get_engine()\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 832, in get_engine\n return self._factory.get_writer_engine()\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 372, in get_writer_engine\n self._start()\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 508, in _start\n self._setup_for_connection(\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 533, in _setup_for_connection\n engine = engines.create_engine(\n File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator\n return wrapped(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 201, in create_engine\n test_conn = _test_connection(engine, max_retries, retry_interval)\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 376, in _test_connection\n return engine.connect()\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2209, in connect\n return self._connection_cls(self, **kwargs)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 103, in __init__\n else engine.raw_connection()\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2306, in raw_connection\n return self._wrap_pool_connect(\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2279, in _wrap_pool_connect\n Connection._handle_dbapi_exception_noconnection(\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1545, in _handle_dbapi_exception_noconnection\n util.raise_from_cause(newraise, exc_info)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb, cause=cause)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise\n raise value.with_traceback(tb)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect\n return fn()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 303, in unique_connection\n return _ConnectionFairy._checkout(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 760, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 492, in checkout\n rec = pool._do_get()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 139, in _do_get\n self._dec_overflow()\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 136, in _do_get\n return self._create_connection()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 308, in _create_connection\n return _ConnectionRecord(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 437, in __init__\n self.__connect(first_connect_check=True)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 639, in __connect\n connection = pool._invoke_creator(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect\n return dialect.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 482, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 94, in Connect\n return Connection(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 325, in __init__\n self.connect()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 598, in connect\n self._get_server_information()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 975, in _get_server_information\n packet = self._read_packet()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet\n packet.check_error()\n File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error\n err.raise_mysql_exception(self._data)\n File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception\n raise errorclass(errno, errval)\noslo_db.exception.DBError: (pymysql.err.InternalError) (2003, "Can\'t connect to remote MySQL server for client connected to \'127.0.0.1:3306\'")\n(Background on this error at: http://sqlalche.me/e/2j85)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect\n return fn()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 303, in unique_connection\n return _ConnectionFairy._checkout(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 760, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 492, in checkout\n rec = pool._do_get()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 139, in _do_get\n self._dec_overflow()\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 136, in _do_get\n return self._create_connection()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 308, in _create_connection\n return _ConnectionRecord(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 437, in __init__\n self.__connect(first_connect_check=True)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 639, in __connect\n connection = pool._invoke_creator(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect\n return dialect.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 482, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 94, in Connect\n return Connection(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 325, in __init__\n self.connect()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 598, in connect\n self._get_server_information()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 975, in _get_server_information\n packet = self._read_packet()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet\n packet.check_error()\n File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error\n err.raise_mysql_exception(self._data)\n File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception\n raise errorclass(errno, errval)\npymysql.err.InternalError: (2003, "Can\'t connect to remote MySQL server for client connected to \'127.0.0.1:3306\'")\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/cmd/manage.py", line 2777, in main\n ret = fn(*fn_args, **fn_kwargs)\n File "/usr/lib/python3/dist-packages/nova/cmd/manage.py", line 697, in sync\n return migration.db_sync(version, database=\'api\')\n File "/usr/lib/python3/dist-packages/nova/db/migration.py", line 26, in db_sync\n return IMPL.db_sync(version=version, database=database, context=context)\n File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/migration.py", line 53, in db_sync\n current_version = db_version(database, context=context)\n File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/migration.py", line 95, in db_version\n return _db_version(repository, database, context)\n File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/migration.py", line 100, in _db_version\n return versioning_api.db_version(get_engine(database, context=context),\n File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/migration.py", line 43, in get_engine\n return db_session.get_api_engine()\n File "/usr/lib/python3/dist-packages/nova/db/sqlalchemy/api.py", line 151, in get_api_engine\n return api_context_manager.writer.get_engine()\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 832, in get_engine\n return self._factory.get_writer_engine()\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 372, in get_writer_engine\n self._start()\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 508, in _start\n self._setup_for_connection(\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 533, in _setup_for_connection\n engine = engines.create_engine(\n File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator\n return wrapped(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 201, in create_engine\n test_conn = _test_connection(engine, max_retries, retry_interval)\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 376, in _test_connection\n return engine.connect()\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2209, in connect\n return self._connection_cls(self, **kwargs)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 103, in __init__\n else engine.raw_connection()\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2306, in raw_connection\n return self._wrap_pool_connect(\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2279, in _wrap_pool_connect\n Connection._handle_dbapi_exception_noconnection(\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1545, in _handle_dbapi_exception_noconnection\n util.raise_from_cause(newraise, exc_info)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb, cause=cause)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise\n raise value.with_traceback(tb)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect\n return fn()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 303, in unique_connection\n return _ConnectionFairy._checkout(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 760, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 492, in checkout\n rec = pool._do_get()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 139, in _do_get\n self._dec_overflow()\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 136, in _do_get\n return self._create_connection()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 308, in _create_connection\n return _ConnectionRecord(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 437, in __init__\n self.__connect(first_connect_check=True)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 639, in __connect\n connection = pool._invoke_creator(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect\n return dialect.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 482, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 94, in Connect\n return Connection(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 325, in __init__\n self.connect()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 598, in connect\n self._get_server_information()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 975, in _get_server_information\n packet = self._read_packet()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet\n packet.check_error()\n File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error\n err.raise_mysql_exception(self._data)\n File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception\n raise errorclass(errno, errval)\noslo_db.exception.DBError: (pymysql.err.InternalError) (2003, "Can\'t connect to remote MySQL server for client connected to \'127.0.0.1:3306\'")\n(Background on this error at: http://sqlalche.me/e/2j85)\n\n'
2020-10-28 22:47:44 INFO juju-log shared-db:18: Creating cell0 database records
2020-10-28 22:47:46 DEBUG shared-db-relation-changed Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
2020-10-28 22:47:47 ERROR juju-log shared-db:18: map_cell0 failed
b'An error has occurred:\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect\n return fn()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 303, in unique_connection\n return _ConnectionFairy._checkout(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 760, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 492, in checkout\n rec = pool._do_get()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 139, in _do_get\n self._dec_overflow()\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 136, in _do_get\n return self._create_connection()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 308, in _create_connection\n return _ConnectionRecord(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 437, in __init__\n self.__connect(first_connect_check=True)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 639, in __connect\n connection = pool._invoke_creator(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect\n return dialect.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 482, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 94, in Connect\n return Connection(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 325, in __init__\n self.connect()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 598, in connect\n self._get_server_information()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 975, in _get_server_information\n packet = self._read_packet()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet\n packet.check_error()\n File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error\n err.raise_mysql_exception(self._data)\n File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception\n raise errorclass(errno, errval)\npymysql.err.InternalError: (2003, "Can\'t connect to remote MySQL server for client connected to \'127.0.0.1:3306\'")\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/cmd/manage.py", line 2777, in main\n ret = fn(*fn_args, **fn_kwargs)\n File "/usr/lib/python3/dist-packages/nova/cmd/manage.py", line 818, in map_cell0\n self._map_cell0(database_connection=database_connection)\n File "/usr/lib/python3/dist-packages/nova/cmd/manage.py", line 852, in _map_cell0\n cell_mapping.create()\n File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 226, in wrapper\n return fn(self, *args, **kwargs)\n File "/usr/lib/python3/dist-packages/nova/objects/cell_mapping.py", line 198, in create\n db_mapping = self._create_in_db(self._context, self.obj_get_changes())\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1013, in wrapper\n with self._transaction_scope(context):\n File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__\n return next(self.gen)\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1058, in _transaction_scope\n with current._produce_block(\n File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__\n return next(self.gen)\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 657, in _session\n self.session = self.factory._create_session(\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 418, in _create_session\n self._start()\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 508, in _start\n self._setup_for_connection(\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 533, in _setup_for_connection\n engine = engines.create_engine(\n File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator\n return wrapped(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 201, in create_engine\n test_conn = _test_connection(engine, max_retries, retry_interval)\n File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/engines.py", line 376, in _test_connection\n return engine.connect()\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2209, in connect\n return self._connection_cls(self, **kwargs)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 103, in __init__\n else engine.raw_connection()\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2306, in raw_connection\n return self._wrap_pool_connect(\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2279, in _wrap_pool_connect\n Connection._handle_dbapi_exception_noconnection(\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1545, in _handle_dbapi_exception_noconnection\n util.raise_from_cause(newraise, exc_info)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 398, in raise_from_cause\n reraise(type(exception), exception, tb=exc_tb, cause=cause)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 152, in reraise\n raise value.with_traceback(tb)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2276, in _wrap_pool_connect\n return fn()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 303, in unique_connection\n return _ConnectionFairy._checkout(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 760, in _checkout\n fairy = _ConnectionRecord.checkout(pool)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 492, in checkout\n rec = pool._do_get()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 139, in _do_get\n self._dec_overflow()\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 153, in reraise\n raise value\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/impl.py", line 136, in _do_get\n return self._create_connection()\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 308, in _create_connection\n return _ConnectionRecord(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 437, in __init__\n self.__connect(first_connect_check=True)\n File "/usr/lib/python3/dist-packages/sqlalchemy/pool/base.py", line 639, in __connect\n connection = pool._invoke_creator(self)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 114, in connect\n return dialect.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 482, in connect\n return self.dbapi.connect(*cargs, **cparams)\n File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 94, in Connect\n return Connection(*args, **kwargs)\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 325, in __init__\n self.connect()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 598, in connect\n self._get_server_information()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 975, in _get_server_information\n packet = self._read_packet()\n File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 684, in _read_packet\n packet.check_error()\n File "/usr/lib/python3/dist-packages/pymysql/protocol.py", line 220, in check_error\n err.raise_mysql_exception(self._data)\n File "/usr/lib/python3/dist-packages/pymysql/err.py", line 109, in raise_mysql_exception\n raise errorclass(errno, errval)\noslo_db.exception.DBError: (pymysql.err.InternalError) (2003, "Can\'t connect to remote MySQL server for client connected to \'127.0.0.1:3306\'")\n(Background on this error at: http://sqlalche.me/e/2j85)\n\n'
2020-10-28 22:47:47 INFO juju-log shared-db:18: Retrying 'migrate_nova_databases' 4 more times (delay=6)
