Glance API v2 suffers from bug 1036193

[Paul Bourke]

It seems glance-api v2 doesn't contain the fix for bug #1036193 and still uses the same ENGINE across workers. This can be demonstrated on devstack by the following crude script:

# Create 100 image entries
for i in $(seq 1 100); do
    glance image-create
done

# <restart the server>

# Then run:
for i in $(seq 1 100); do
    glance --os-image-api-version=2 image-list &
done

You'll get the errors outlined by David at https://bugs.launchpad.net/glance/+bug/1036193/comments/1

[Mark Washenberger (markwash)]

I have tested this on devstack with workers = 2, 4, 8 and have not been able to reproduce the problem. In addition, it appears the fix to bug #1036193 (namely https://review.openstack.org/#/c/11612/) would affect everything that happens in the glance-api wsgi app, which covers both v1 and v2.

Can you help me understand why I'm not seeing this bug? I followed the instructions you posted.

[Paul Bourke (pauldbourke)]

Hi Mark,

I should also have referenced bug #1089916, which was a follow up to #1036193.

From what I can see the problem is in glance/gateway.py, as it calls 'self.db_api.configure_db()' in it's __init__ whereas it should probably be calling setup_db_env() to avoid initialising the engine before the fork.

Unsure as to why you're unable to reproduce, I still seem to be able to consitently with 4 api workers. Would be useful if perhaps someone else could try?

[Mark Washenberger (markwash)]

I haven't tried to reproduce it again, but just out of curiousity can you see if this patch helps things out?

diff --git a/glance/gateway.py b/glance/gateway.py
index a93e7c3..f42a26c 100644
--- a/glance/gateway.py
+++ b/glance/gateway.py
@@ -25,7 +25,7 @@ class Gateway(object):
     def __init__(self, db_api=None, store_api=None, notifier=None,
                  policy_enforcer=None):
         self.db_api = db_api or glance.db.get_api()
- self.db_api.configure_db()
+ self.db_api.setup_db_env()
         self.store_api = store_api or glance.store
         self.notifier = notifier or glance.notifier.Notifier()
         self.policy = policy_enforcer or policy.Enforcer()

[Paul Bourke (pauldbourke)]

Yes from some basic testing that change does appear to fix it.

[Mark Washenberger (markwash)]

Great, I'll propose a fix!

[Mark Washenberger (markwash)]

Also, apparently I read your earlier message selectively, and missed out on the fact that you had already triaged the problem :-)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/25192

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25192
Committed: http://github.com/openstack/glance/commit/ba7c059c9e0971ff7be6dd2f2fc3861140ae8011
Submitter: Jenkins
Branch: master

commit ba7c059c9e0971ff7be6dd2f2fc3861140ae8011
Author: Mark J. Washenberger <email address hidden>
Date: Fri Mar 22 15:14:11 2013 -0700

    Fix domain database initialization

    To fix bug 1089916, calls to db_api.configure_db() were changed to
    setup_db_env(). However, the gateway domain initialization didn't get
    the memo. This patch makes sure setup_db_env() is called in all the
    relevant places.

    Fixes bug 1131705.

    Change-Id: Ieec141192b8a5a045dadba6d70abaa25748b85b2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/25636

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25636
Committed: http://github.com/openstack/glance/commit/eb21c4ec9f262043f49c18e48fc6ad728e1f6189
Submitter: Jenkins
Branch: milestone-proposed

commit eb21c4ec9f262043f49c18e48fc6ad728e1f6189
Author: Mark J. Washenberger <email address hidden>
Date: Fri Mar 22 15:14:11 2013 -0700

    Fix domain database initialization

    To fix bug 1089916, calls to db_api.configure_db() were changed to
    setup_db_env(). However, the gateway domain initialization didn't get
    the memo. This patch makes sure setup_db_env() is called in all the
    relevant places.

    Fixes bug 1131705.

    Change-Id: Ieec141192b8a5a045dadba6d70abaa25748b85b2