microk8s

Bug #2033680
Comment #6

Comment 6 for bug 2033680

Revision history for this message

Peter Jose De Sousa (pjds) wrote on 2024-07-04:

hitting this issue following quickstart guide at https://ubuntu.com/openstack/install - my nova unit and cinder-ceph unit are both blocked. Cinder-ceph pod logging does not reveal much, although I have the failure to resolve mysql in nova.

qlalchemy.engines [-] SQL connection failed. 10 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b
[00m\n2024-07-03 14:04:24.636 149 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 9 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.lo
cal\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:04:34.659 149 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 8 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server o
n \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:04:44.684 149 WARNING oslo_db.sqlalchemy.engines [-] SQL connect' [truncated], stderr=''
2024-07-03T14:05:55.666Z [container-agent] 2024-07-03 14:05:55 WARNING juju-log identity-service:124: Finished call to 'ops_sunbeam.charm.OSBaseOperatorCharmK8S._retry_db_sync' after 986.756(s), this was the 8th time calling it.
2024-07-03T14:09:47.104Z [container-agent] 2024-07-03 14:09:47 WARNING juju-log identity-service:124: DB Sync pebble exec error: non-zero exit code 255 executing ['sudo', '-u', 'nova', 'nova-manage', 'api_db', 'sync'], stdout='2024-07-03 14:08:06.043 152 WARNING oslo_db.s
qlalchemy.engines [-] SQL connection failed. 10 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b
[00m\n2024-07-03 14:08:16.070 152 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 9 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.lo
cal\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:08:26.098 152 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 8 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server o
n \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:08:36.123 152 WARNING oslo_db.sqlalchemy.engines [-] SQL connect' [truncated], stderr=''
2024-07-03T14:09:47.115Z [container-agent] 2024-07-03 14:09:47 WARNING juju-log identity-service:124: Finished call to 'ops_sunbeam.charm.OSBaseOperatorCharmK8S._retry_db_sync' after 1218.204(s), this was the 9th time calling it.
2024-07-03T14:15:46.669Z [container-agent] 2024-07-03 14:15:46 WARNING juju-log identity-service:124: DB Sync pebble exec error: non-zero exit code 255 executing ['sudo', '-u', 'nova', 'nova-manage', 'api_db', 'sync'], stdout='2024-07-03 14:14:05.668 155 WARNING oslo_db.s
qlalchemy.engines [-] SQL connection failed. 10 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b
[00m\n2024-07-03 14:14:15.695 155 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 9 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.lo
cal\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:14:25.721 155 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 8 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server o
n \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:14:35.739 155 WARNING oslo_db.sqlalchemy.engines [-] SQL connect' [truncated], stderr=''
2024-07-03T14:15:46.679Z [container-agent] 2024-07-03 14:15:46 WARNING juju-log identity-service:124: Finished call to 'ops_sunbeam.charm.OSBaseOperatorCharmK8S._retry_db_sync' after 1577.769(s), this was the 10th time calling it.
2024-07-03T14:15:46.690Z [container-agent] 2024-07-03 14:15:46 WARNING juju-log identity-service:124: Charm is blocked in section 'Bootstrapping' due to 'DB sync failed'
2024-07-03T14:15:50.745Z [container-agent] 2024-07-03 14:15:50 INFO juju.worker.uniter.operation runhook.go:186 ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)
2024-07-03T14:15:56.990Z [container-agent] 2024-07-03 14:15:56 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)
2024-07-03T14:20:56.522Z [container-agent] 2024-07-03 14:20:56 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)

Unit                         Workload  Agent  Address     Ports  Message
certificate-authority/0*     active    idle   10.1.2.199         
cinder-ceph-mysql-router/0*  active    idle   10.1.2.224         
cinder-ceph/0*               blocked   idle   10.1.2.226         (ceph) integration missing
cinder-mysql-router/0*       active    idle   10.1.2.217         
cinder-mysql/0*              active    idle   10.1.2.229         Primary
cinder/0*                    active    idle   10.1.2.225         
glance-mysql-router/0*       active    idle   10.1.2.221         
glance-mysql/0*              active    idle   10.1.2.220         Primary
glance/0*                    active    idle   10.1.2.237         
horizon-mysql-router/0*      active    idle   10.1.2.228         
horizon-mysql/0*             active    idle   10.1.2.236         Primary
horizon/0*                   active    idle   10.1.2.233         
keystone-mysql-router/0*     active    idle   10.1.2.235         
keystone-mysql/0*            active    idle   10.1.2.210         Primary
keystone/0*                  active    idle   10.1.2.241         
neutron-mysql-router/0*      active    idle   10.1.2.234         
neutron-mysql/0*             active    idle   10.1.2.230         Primary
neutron/0*                   active    idle   10.1.2.242         
nova-api-mysql-router/0*     active    idle   10.1.2.243         
nova-cell-mysql-router/0*    active    idle   10.1.2.238         
nova-mysql-router/0*         active    idle   10.1.2.223         
nova-mysql/0*                active    idle   10.1.2.232         Primary
nova/0*                      blocked   idle   10.1.2.231         (workload) DB sync failed
ovn-central/0*               active    idle   10.1.2.214         
ovn-relay/0*                 active    idle   10.1.2.202         
placement-mysql-router/0*    active    idle   10.1.2.215         
placement-mysql/0*           active    idle   10.1.2.212         Primary
placement/0*                 active    idle   10.1.2.222         
rabbitmq/0*                  active    idle   10.1.2.209         
traefik-public/0*            active    idle   10.1.2.206         
traefik/0*                   active    idle   10.1.2.218

Offer                  Application            Charm                     Rev  Connected  Endpoint              Interface             Role
cert-distributor       keystone               keystone-k8s              195  0/0        send-ca-cert          certificate_transfer  provider
certificate-authority  certificate-authority  self-signed-certificates  151  0/0        certificates          tls-certificates      provider
cinder-ceph            cinder-ceph            cinder-ceph-k8s           77   0/0        ceph-access           cinder-ceph-key       provider
keystone-credentials   keystone               keystone-k8s              195  0/0        identity-credentials  keystone-credentials  provider
keystone-endpoints     keystone               keystone-k8s              195  0/0        identity-service      keystone              provider
nova                   nova                   nova-k8s                  90   0/0        nova-service          nova                  provider
ovn-relay              ovn-relay              ovn-relay-k8s             79   0/0        ovsdb-cms-relay       ovsdb-cms             provider
rabbitmq               rabbitmq               rabbitmq-k8s              33   0/0        amqp                  rabbitmq              provider

qlalchemy.engines [-] SQL connection failed. 10 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b
[00m\n2024-07-03 14:04:24.636 149 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 9 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.lo
cal\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:04:34.659 149 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 8 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server o
n \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:04:44.684 149 WARNING oslo_db.sqlalchemy.engines [-] SQL connect' [truncated], stderr=''
2024-07-03T14:05:55.666Z [container-agent] 2024-07-03 14:05:55 WARNING juju-log identity-service:124: Finished call to 'ops_sunbeam.charm.OSBaseOperatorCharmK8S._retry_db_sync' after 986.756(s), this was the 8th time calling it.                                            
2024-07-03T14:09:47.104Z [container-agent] 2024-07-03 14:09:47 WARNING juju-log identity-service:124: DB Sync pebble exec error: non-zero exit code 255 executing ['sudo', '-u', 'nova', 'nova-manage', 'api_db', 'sync'], stdout='2024-07-03 14:08:06.043 152 WARNING oslo_db.s
qlalchemy.engines [-] SQL connection failed. 10 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b
[00m\n2024-07-03 14:08:16.070 152 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 9 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.lo
cal\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:08:26.098 152 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 8 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server o
n \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:08:36.123 152 WARNING oslo_db.sqlalchemy.engines [-] SQL connect' [truncated], stderr=''
2024-07-03T14:09:47.115Z [container-agent] 2024-07-03 14:09:47 WARNING juju-log identity-service:124: Finished call to 'ops_sunbeam.charm.OSBaseOperatorCharmK8S._retry_db_sync' after 1218.204(s), this was the 9th time calling it.
2024-07-03T14:15:46.669Z [container-agent] 2024-07-03 14:15:46 WARNING juju-log identity-service:124: DB Sync pebble exec error: non-zero exit code 255 executing ['sudo', '-u', 'nova', 'nova-manage', 'api_db', 'sync'], stdout='2024-07-03 14:14:05.668 155 WARNING oslo_db.s
qlalchemy.engines [-] SQL connection failed. 10 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b
[00m\n2024-07-03 14:14:15.695 155 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 9 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server on \'nova-api-mysql-router.openstack.svc.cluster.lo
cal\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:14:25.721 155 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. 8 attempts left.: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can\'t connect to MySQL server o
n \'nova-api-mysql-router.openstack.svc.cluster.local\' ([Errno -2] Name or service not known)")\x1b[00m\n2024-07-03 14:14:35.739 155 WARNING oslo_db.sqlalchemy.engines [-] SQL connect' [truncated], stderr=''                                                                
2024-07-03T14:15:46.679Z [container-agent] 2024-07-03 14:15:46 WARNING juju-log identity-service:124: Finished call to 'ops_sunbeam.charm.OSBaseOperatorCharmK8S._retry_db_sync' after 1577.769(s), this was the 10th time calling it.                                          
2024-07-03T14:15:46.690Z [container-agent] 2024-07-03 14:15:46 WARNING juju-log identity-service:124: Charm is blocked in section 'Bootstrapping' due to 'DB sync failed'                                       
2024-07-03T14:15:50.745Z [container-agent] 2024-07-03 14:15:50 INFO juju.worker.uniter.operation runhook.go:186 ran "identity-service-relation-changed" hook (via hook dispatching script: dispatch)                                
2024-07-03T14:15:56.990Z [container-agent] 2024-07-03 14:15:56 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)                                                                                                
2024-07-03T14:20:56.522Z [container-agent] 2024-07-03 14:20:56 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)