Retry mechanism does not work on startup when used with MySQL
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Invalid
|
High
|
Ann Taraday | ||
oslo.db |
Fix Released
|
High
|
Roman Podoliaka | ||
Juno |
Fix Released
|
High
|
Roman Podoliaka |
Bug Description
This is initially revealed as Red Hat bug: https:/
The problem shows up when Neutron or any other oslo.db based projects start while MySQL server is not up yet. Instead of retrying connection as per max_retries and retry_interval, service just crashes with return code 1.
This is because during engine initialization, "engine.
This behaviour was checked for both oslo.db 0.4.0 and 1.0.1.
I suspect this is a regression from the original db code from oslo-incubator though I haven't checked it specifically.
The easiest way to reproduce the traceback is:
1. stop MariaDB.
2. execute the following Python script:
'''
import oslo.db.
url = 'mysql:
engine = oslo.db.
'''
The following traceback can be seen in service log:
2014-10-01 13:46:10.588 5812 TRACE neutron Traceback (most recent call last):
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/bin/
2014-10-01 13:46:10.588 5812 TRACE neutron sys.exit(main())
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron neutron_api = service.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron LOG.exception(
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron six.reraise(
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron service.start()
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron self.wsgi_app = _run_wsgi(
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron app = config.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron app = deploy.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return loadobj(APP, uri, name=name, **kw)
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return context.create()
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return self.object_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron **context.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron val = callable(*args, **kw)
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron app = loader.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron name=name, global_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return self.object_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron **context.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron val = callable(*args, **kw)
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron app = loader.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron name=name, global_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return self.object_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return fix_call(
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron val = callable(*args, **kw)
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return cls(**local_config)
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron plugin = manager.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return weakref.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron cls._create_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return f(*args, **kwargs)
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron cls._instance = cls()
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron plugin_provider)
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return plugin_class()
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron self.type_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron driver.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron self._sync_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron session = db_api.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron facade = _create_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron _FACADE = session.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron retry_interval=
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron **engine_kwargs)
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron connection_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron self._url_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return self._dispatch_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron if self._invoke_fn(fn, arg, kw) is not None:
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron return fn(*arg, **kw)
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib/
2014-10-01 13:46:10.588 5812 TRACE neutron realmode = engine.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron connection = self.contextual
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron self.pool.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron return _ConnectionFair
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron fairy = _ConnectionReco
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron rec = pool._do_get()
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron return self._create_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron return _ConnectionReco
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron self.connection = self.__connect()
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron connection = self.__
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron connection_
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron reraise(
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron return dialect.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron return self.dbapi.
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron return Connection(*args, **kwargs)
2014-10-01 13:46:10.588 5812 TRACE neutron File "/usr/lib64/
2014-10-01 13:46:10.588 5812 TRACE neutron super(Connection, self)._
2014-10-01 13:46:10.588 5812 TRACE neutron OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '10.35.161.235' (111)") None None
Changed in oslo.db: | |
status: | New → Triaged |
importance: | Undecided → High |
Changed in oslo.db: | |
assignee: | nobody → Roman Podoliaka (rpodolyaka) |
status: | Triaged → In Progress |
Changed in neutron: | |
importance: | Undecided → Medium |
status: | New → Confirmed |
importance: | Medium → High |
Changed in oslo.db: | |
milestone: | none → next-kilo |
tags: | added: icehouse-backport-potential |
tags: | added: juno-rc-potential |
tags: |
added: juno-backport-potential removed: juno-rc-potential |
Changed in oslo.db: | |
status: | Fix Committed → Fix Released |
Changed in neutron: | |
assignee: | nobody → Ann Kamyshnikova (akamyshnikova) |
tags: | added: db |
This bug affects neutron since after a reboot it might be possible that the service will fail at startup if it's started before the mysql service.