OpenStack Bundles

After deployment keep getting varous Mysql connection issues #58

Bug #1813143 reported by Nick Edwards on 2019-01-24

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Bundles	Expired	Undecided	Unassigned

Bug Description

Constant connection errors like the following:

2019-01-24 10:00:45.751 85135 ERROR oslo_service.periodic_task oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.192.10' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-24 10:00:45.751 85135 ERROR oslo_service.periodic_task
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task [req-2b010b4f-657b-489a-aa4d-56bf2a4af2cf - - - - -] Error during SchedulerManager._discover_hosts_in_cells: oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.192.10' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task Traceback (most recent call last):
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 920, in connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task **kwargs)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/eventlet/green/socket.py", line 69, in create_connection
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task raise err
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/eventlet/green/socket.py", line 59, in create_connection
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task sock.connect(sa)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 256, in connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task socket_checkerr(fd)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/eventlet/greenio/base.py", line 46, in socket_checkerr
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task raise socket.error(err, errno.errorcode[err])
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task ConnectionRefusedError: [Errno 111] ECONNREFUSED
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task During handling of the above exception, another exception occurred:
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task Traceback (most recent call last):
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return fn()
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 403, in connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return _ConnectionFairy._checkout(self)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 791, in _checkout
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task fairy = _ConnectionRecord.checkout(pool)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 537, in checkout
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task rec._checkin_failed(err)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task compat.reraise(exc_type, exc_value, exc_tb)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 187, in reraise
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task raise value
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 534, in checkout
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task dbapi_connection = rec.get_connection()
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 630, in get_connection
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task self.__connect()
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 674, in __connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task connection = pool._invoke_creator(self)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 106, in connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return dialect.connect(*cargs, **cparams)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 411, in connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return self.dbapi.connect(*cargs, **cparams)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 90, in Connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return Connection(*args, **kwargs)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 699, in __init__
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task self.connect()
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 967, in connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task raise exc
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on '192.168.192.10' ([Errno 111] ECONNREFUSED)")
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task The above exception was the direct cause of the following exception:
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task Traceback (most recent call last):
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task task(self, context)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 71, in _discover_hosts_in_cells
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task host_mappings = host_mapping_obj.discover_hosts(context)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/nova/objects/host_mapping.py", line 248, in discover_hosts
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task cell_mappings = objects.CellMappingList.get_all(ctxt)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task result = fn(cls, context, *args, **kwargs)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/nova/objects/cell_mapping.py", line 257, in get_all
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task db_mappings = cls._get_all_from_db(context)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 993, in wrapper
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return fn(*args, **kwargs)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/nova/objects/cell_mapping.py", line 253, in _get_all_from_db
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task asc(api_models.CellMapping.id)).all()
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2783, in all
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return list(self)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2935, in __iter__
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return self._execute_and_instances(context)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2956, in _execute_and_instances
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task close_with_result=True)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2965, in _get_bind_args
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task **kw
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/orm/query.py", line 2947, in _connection_from_session
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task conn = self.session.connection(**kw)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 1035, in connection
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task execution_options=execution_options)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task engine, execution_options)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task conn = bind.contextual_connect()
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task self._wrap_pool_connect(self.pool.connect, None),
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task e, dialect, self)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task util.raise_from_cause(newraise, exc_info)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task reraise(type(exception), exception, tb=exc_tb, cause=cause)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 186, in reraise
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task raise value.with_traceback(tb)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return fn()
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 403, in connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return _ConnectionFairy._checkout(self)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 791, in _checkout
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task fairy = _ConnectionRecord.checkout(pool)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 537, in checkout
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task rec._checkin_failed(err)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task compat.reraise(exc_type, exc_value, exc_tb)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/util/compat.py", line 187, in reraise
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task raise value
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 534, in checkout
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task dbapi_connection = rec.get_connection()
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 630, in get_connection
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task self.__connect()
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/pool.py", line 674, in __connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task connection = pool._invoke_creator(self)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/engine/strategies.py", line 106, in connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return dialect.connect(*cargs, **cparams)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sqlalchemy/engine/default.py", line 411, in connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return self.dbapi.connect(*cargs, **cparams)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/pymysql/__init__.py", line 90, in Connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task return Connection(*args, **kwargs)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 699, in __init__
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task self.connect()
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/pymysql/connections.py", line 967, in connect
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task raise exc
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '192.168.192.10' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8)
2019-01-24 10:00:46.851 85147 ERROR oslo_service.periodic_task

This does not happen with version 57 of the bundle.

Nick Edwards (njwedwards-3) on 2019-01-24

summary:

- After deployment keep getting varous Mysql connection issues
+ After deployment keep getting varous Mysql connection issues #58

Revision history for this message

Frode Nordahl (fnordahl) wrote on 2019-01-29:

What is the state of the MySQL daemon on the mysql unit? Does the logs there reveal any information about this state? Please examine /var/log/juju/unit-mysql-0.log and /var/log/mysql/error.log. Examining timestamps for start and stop events for the mysql service in /var/log/syslog might also be useful.

Changed in openstack-bundles:
status:	New → Incomplete

Revision history for this message

Nick Edwards (njwedwards-3) wrote on 2019-01-30:

I have since had to go back to #57 and I do not get Mysql issues. There definitely seems to be something wrong in #58 but I will have to try and deploy again then update.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2019-04-01:

[Expired for OpenStack Bundles because there has been no activity for 60 days.]

Changed in openstack-bundles:
status:	Incomplete → Expired

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.