keystone upgrade fails due to mysql not being available when both installed on the same server

[Jamie Strandboge]

I was upgrading a 12.04 VM and encountered the following error on upgrade:

Setting up keystone (2012.1+stable~20120608-aff45d6-0ubuntu1) ...
Traceback (most recent call last):
  File "/usr/bin/keystone-manage", line 28, in <module>
    cli.main(argv=sys.argv, config_files=config_files)
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 148, in main
    return run(cmd, (args[:1] + args[2:]))
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 134, in run
    return CMDS[cmd](argv=args).run()
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 36, in run
    return self.main()
  File "/usr/lib/python2.7/dist-packages/keystone/cli.py", line 57, in main
    driver.db_sync()
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/sql.py", line 135, in db_sync
    migration.db_sync()
  File "/usr/lib/python2.7/dist-packages/keystone/common/sql/migration.py", line 50, in db_sync
    current_version = db_version()
  File "/usr/lib/python2.7/dist-packages/keystone/common/sql/migration.py", line 66, in db_version
    return db_version_control(0)
  File "/usr/lib/python2.7/dist-packages/keystone/common/sql/migration.py", line 72, in db_version_control
    CONF.sql.connection, repo_path, version)
  File "<string>", line 2, in version_control
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/util/__init__.py", line 159, in with_engine
    return f(*a, **kw)
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/api.py", line 250, in version_control
    ControlledSchema.create(engine, repository, version)
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 139, in create
    table = cls._create_table_version(engine, repository, version)
  File "/usr/lib/python2.7/dist-packages/migrate/versioning/schema.py", line 180, in _create_table_version
    if not table.exists():
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/schema.py", line 549, in exists
    self.name, schema=self.schema)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2274, in run_callable
    conn = self.contextual_connect()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2340, in contextual_connect
    self.pool.connect(),
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 210, in connect
    return _ConnectionFairy(self).checkout()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 371, in __init__
    rec = self._connection_record = pool._do_get()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 697, in _do_get
    con = self._create_connection()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 174, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 256, in __init__
    self.connection = self.__connect()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 316, in __connect
    connection = self.__pool._creator()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 80, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 280, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (OperationalError) (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)") None None
dpkg: error processing keystone (--configure):
 subprocess installed post-installation script returned error exit status 1

This happens when mysql server is upgraded at the same time as keystone due to the ordering of the package upgrade:
...
Preparing to replace mysql-server-5.5 5.5.22-0ubuntu1 (using .../mysql-server-5.5_5.5.24-0ubuntu0.12.04.1_amd64.deb) ...
mysql stop/waiting
...
Preparing to replace keystone 2012.1-0ubuntu1 (using .../keystone_2012.1+stable~20120608-aff45d6-0ubuntu1_all.deb) ...
keystone stop/waiting
Unpacking replacement keystone ...
Preparing to replace python-keystone 2012.1-0ubuntu1 (using .../python-keystone_2012.1+stable~20120608-aff45d6-0ubuntu1_all.deb) ...
Unpacking replacement python-keystone ...
...
Setting up keystone (2012.1+stable~20120608-aff45d6-0ubuntu1) ...
Traceback (most recent call last):
... sqlalchemy.exc.OperationalError: (OperationalError) (2002, "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)") None None
dpkg: error processing keystone (--configure):
 subprocess installed post-installation script returned error exit status 1
...
Setting up mysql-server-5.5 (5.5.24-0ubuntu0.12.04.1) ...
mysql start/running, process 29811

Notice how mysql server is stopped, keystone is upgraded with its postinst running before mysql server is started again.

'sudo apt-get -f install' will resolve the issue (of course).

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug. I would mark it high priority, as apt-get -f install is a workaround I'll lower that to medium.

[David Kranz (david-kranz)]

I just ran into this after 'apt-get update' and 'apt-get upgrade' to get the new stable versions of Folsom. IMO, if those two steps are the "endorsed" way to do that upgrade, I don't see how this bug could be considered not high priority.

[James Page (james-page)]

This is a specific issue when both mysql and keystone are installed on the same server; apt has no way of knowing that mysql must be running for the keystone update to successfully configure.