glance-api with workers !=0 causes mysql issues

[David Kranz]

The problem is that configure_db is called before the workers are forked. This causes all of the workers to share the same underlying socket connection to the database even though each process has its own copy of ENGINE. I bet a stress test blasting at glance-api would fail quickly. I didn't actually do such a stress test on glance but I did on another application that borrowed this part of the glance code and found the same problem. I believe this could cause arbitrary corruption of the glance database.

 Setting workers to 8:

stack@xg08:~/devstack$ ps aux | grep glance-api
root 32691 0.1 0.0 204212 35000 ? Ss 09:33 0:00 /usr/bin/python /usr/local/bin/glance-api
root 32697 0.0 0.0 204492 30744 ? S 09:33 0:00 /usr/bin/python /usr/local/bin/glance-api
root 32698 0.0 0.0 204496 30744 ? S 09:33 0:00 /usr/bin/python /usr/local/bin/glance-api
root 32699 0.0 0.0 204496 30744 ? S 09:33 0:00 /usr/bin/python /usr/local/bin/glance-api
root 32700 0.0 0.0 204496 30744 ? S 09:33 0:00 /usr/bin/python /usr/local/bin/glance-api
root 32701 0.0 0.0 204496 30740 ? S 09:33 0:00 /usr/bin/python /usr/local/bin/glance-api
root 32702 0.0 0.0 204496 30744 ? S 09:33 0:00 /usr/bin/python /usr/local/bin/glance-api
root 32703 0.0 0.0 204496 30744 ? S 09:33 0:00 /usr/bin/python /usr/local/bin/glance-api
root 32704 0.0 0.0 204496 30744 ? S 09:33 0:00 /usr/bin/python /usr/local/bin/glance-api

stack@xg08:~/devstack$ sudo lsof -p 14562 | grep mysqld.sock
mysqld 14562 mysql 12u unix 0xffff880beebcf500 0t0 29258 /var/run/mysqld/mysqld.sock
mysqld 14562 mysql 31u unix 0xffff880bec75c780 0t0 2356782 /var/run/mysqld/mysqld.sock

[David Kranz (david-kranz)]

Putting a printf in configure_db also showed it was getting called only in the initial process. Here is one error I saw in the other app before fixing this:

  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 1947, in all
    return list(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2057, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2072, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1405, in execute
    params)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1538, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1646, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1639, in _execute_context
    context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 330, in do_execute
    cursor.execute(statement, parameters)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
OperationalError: (OperationalError) (2006, 'MySQL server has gone away') 'SELECT files.id AS files_id, files.name AS file\
s_name, files.description AS files_description, files.command_line AS files_command_line, files.jobs_id AS files_jobs_id, \
files.content_type AS files_content_type, files.creation_time AS files_creation_time, files.user AS files_user, files.swif\
t_container_name AS files_swift_container_name, files.swift_object_name AS files_swift_object_name \nFROM files \nWHERE fi\
les.name = %s' ('657e9377-1d8c-4d26-bb65-1ab019d33b7c',)

[David Kranz (david-kranz)]

The answer at the bottom of http://serverfault.com/questions/407612/error-2006-mysql-server-has-gone-away is what clued me in to this problem and its solution.

[Jay Pipes (jaypipes)]

Thanks David!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/11612

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/11612
Committed: http://github.com/openstack/glance/commit/112fc71ba453c2f43ce5ccb91108a504bcd6b5b6
Submitter: Jenkins
Branch: master

commit 112fc71ba453c2f43ce5ccb91108a504bcd6b5b6
Author: Brian Waldon <email address hidden>
Date: Sat Aug 18 16:50:15 2012 -0400

    Instantiate wsgi app for each worker

    A single WSGI app, and therefore a single sqlalchemy engine (mysql
    connection), was being used across all API workers. This patch
    instantiates the WSGI app once per worker rather than up front to
    give a separate database connection to each worker.

    Fixes bug 1036193

    Change-Id: I2cbb61fd88a6ec4fb03fa84196ba0a380b3842d0