Non-deterministic results using simple.db api backend and multiple workers

[Timothy Symanczyk]

To reproduce :
1) In glance-api.conf, set "data_api = glance.db.simple.api"

2) Again in glance-api.conf, set "workers = 2" (or higher)

3) Restart glance-api to empty all tables

4) Using any method, create ten public images

5) Using any method, invoke "image list" several times

6) Observe that the results are not consistent between all invocations

--------------

timothy_symanczyk@devstack:~$ openstack image list
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils

timothy_symanczyk@devstack:~$ openstack image create 1
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
(snip)
| name | 1 |
(snip)
+------------------+------------------------------------------------------+
timothy_symanczyk@devstack:~$ openstack image create 2
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
(snip)
| name | 2 |
(snip)
+------------------+------------------------------------------------------+
timothy_symanczyk@devstack:~$ openstack image create 3
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
(snip)
| name | 3 |
(snip)
+------------------+------------------------------------------------------+
timothy_symanczyk@devstack:~$ openstack image create 4
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
(snip)
| name | 4 |
(snip)
+------------------+------------------------------------------------------+
timothy_symanczyk@devstack:~$ openstack image create 5
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
(snip)
| name | 5 |
(snip)
+------------------+------------------------------------------------------+
timothy_symanczyk@devstack:~$ openstack image create 6
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
(snip)
| name | 6 |
(snip)
+------------------+------------------------------------------------------+
timothy_symanczyk@devstack:~$ openstack image create 7
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
(snip)
| name | 7 |
(snip)
+------------------+------------------------------------------------------+
timothy_symanczyk@devstack:~$ openstack image create 8
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
(snip)
| name | 8 |
(snip)
+------------------+------------------------------------------------------+
timothy_symanczyk@devstack:~$ openstack image create 9
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
(snip)
| name | 9 |
(snip)
+------------------+------------------------------------------------------+
timothy_symanczyk@devstack:~$ openstack image create 10
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
(snip)
| name | 10 |
(snip)
+------------------+------------------------------------------------------+
timothy_symanczyk@devstack:~$ openstack image list
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+--------------------------------------+------+--------+
| ID | Name | Status |
+--------------------------------------+------+--------+
| bb8160ee-16ff-41a2-8889-832ef93e0e91 | 10 | queued |
| 6937f3df-9ede-453d-a80a-08bacde838f5 | 9 | queued |
| 258ec9de-e7cb-45fc-b8fb-df6dd141412d | 8 | queued |
| c86e3182-e177-499b-a120-a0249f46b7c5 | 5 | queued |
| ed0bf158-3374-4456-a0ea-607d5e3d1e11 | 4 | queued |
| 41aacad5-dbc3-4175-8c25-b486bb96a478 | 3 | queued |
| 609c9ec4-dfee-4aee-a3c5-cb5229667a85 | 1 | queued |
+--------------------------------------+------+--------+
timothy_symanczyk@devstack:~$ openstack image list
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+--------------------------------------+------+--------+
| ID | Name | Status |
+--------------------------------------+------+--------+
| bb8160ee-16ff-41a2-8889-832ef93e0e91 | 10 | queued |
| 6937f3df-9ede-453d-a80a-08bacde838f5 | 9 | queued |
| 258ec9de-e7cb-45fc-b8fb-df6dd141412d | 8 | queued |
| c86e3182-e177-499b-a120-a0249f46b7c5 | 5 | queued |
| ed0bf158-3374-4456-a0ea-607d5e3d1e11 | 4 | queued |
| 41aacad5-dbc3-4175-8c25-b486bb96a478 | 3 | queued |
| 609c9ec4-dfee-4aee-a3c5-cb5229667a85 | 1 | queued |
+--------------------------------------+------+--------+
timothy_symanczyk@devstack:~$ openstack image list
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+--------------------------------------+------+--------+
| ID | Name | Status |
+--------------------------------------+------+--------+
| f7b4fcc5-36ea-45c3-bd94-f2c179a9f1c8 | 7 | queued |
| a3402651-5879-425c-8029-89e987dd13e8 | 6 | queued |
| 5fade950-723c-404b-87b6-406f413e5378 | 2 | queued |
+--------------------------------------+------+--------+
timothy_symanczyk@devstack:~$ openstack image list
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+--------------------------------------+------+--------+
| ID | Name | Status |
+--------------------------------------+------+--------+
| f7b4fcc5-36ea-45c3-bd94-f2c179a9f1c8 | 7 | queued |
| a3402651-5879-425c-8029-89e987dd13e8 | 6 | queued |
| 5fade950-723c-404b-87b6-406f413e5378 | 2 | queued |
+--------------------------------------+------+--------+
timothy_symanczyk@devstack:~$ openstack image list
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+--------------------------------------+------+--------+
| ID | Name | Status |
+--------------------------------------+------+--------+
| bb8160ee-16ff-41a2-8889-832ef93e0e91 | 10 | queued |
| 6937f3df-9ede-453d-a80a-08bacde838f5 | 9 | queued |
| 258ec9de-e7cb-45fc-b8fb-df6dd141412d | 8 | queued |
| c86e3182-e177-499b-a120-a0249f46b7c5 | 5 | queued |
| ed0bf158-3374-4456-a0ea-607d5e3d1e11 | 4 | queued |
| 41aacad5-dbc3-4175-8c25-b486bb96a478 | 3 | queued |
| 609c9ec4-dfee-4aee-a3c5-cb5229667a85 | 1 | queued |
+--------------------------------------+------+--------+
timothy_symanczyk@devstack:~$

[Timothy Symanczyk (timothy-symanczyk)]

Since glance.db.simple.api is an in-memory backend, a plausible explanation of the above behaviour may be that each individual working is creating and utilizing its own individual backend rather than sharing.

[Cyril Roelandt (cyril-roelandt)]

Is the glance.db.simple.api backend used in production? If not, what is the use case for this backend?

I wonder whether we should fix this (which would involve some complex memory-sharing between WSGI processes) or just document that this backend cannot be used with more than one worker. What do you think?

[Timothy Symanczyk (timothy-symanczyk)]

glance.db.simple.api, to the best of my understanding, is primarily used for local testing. I do not believe it is appropriate for production use.

My initial thoughts were perhaps logging a warning at startup if workers>1 and glance.db.simple.api is detected.

Agreed that a full code fix would be overkill. I'd be totally comfortable with a doc fix.

[Nikhil Komawar (nikhil-komawar)]

It's made for local testing only. Though, one could be trying out a test bed for openstack for say running downstream (or even upstream tests) where the DB load doesn't matter. For example, one could use it to test load on the configured glance_store backends rather than image listing.

In such a case, I was thinking we'd use memcached. But we'd be extra careful for setting memcached by default with simple api as that will make the functional tests setup slower.

So, first step looks like some docs, actually I was about to see if the configuration docs here [1] can be updated to have as section on the data access api. We could propose a lite spec for more resilient use of the simple api.

Anyone working on this actively? If so keep yourself assigned to the bug.

[1] http://docs.openstack.org/developer/glance/configuring.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/365738

[Erno Kuvaja (jokke)]

Hi Cyril,

Thanks for looking into this. I think we have two possibilities to reasonably address this bug:
1) we prevent the process to start if db.simple and workers <1 is configured
2) we override the workers setting to 1 if db.simple is configured and log warning about it

In both cases good documentation is needed explaining why and how this is working on decided manner. We would also want to explain why one should not run multiple glance nodes in such cases.

[Cyril Roelandt (cyril-roelandt)]

@Erno: OK, I like possibility #1 better. I've just updated my patch :)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/365738
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=0cea875293c4279682063e4c45c2cea2b1201ddf
Submitter: Jenkins
Branch: master

commit 0cea875293c4279682063e4c45c2cea2b1201ddf
Author: Cyril Roelandt <email address hidden>
Date: Mon Sep 5 18:12:04 2016 +0200

    Limit workers to 0 or 1 when using db.simple.api

    When using more than one worker, only one of them will return data, thus
    leading users to not have access to every single piece of data at once.

    Change-Id: I0e7e0f0ff7821465a74eb284e8401ddd7238f6df
    Partial-Bug: #1619508
    Co-Authored-By: Cyril Roelandt <email address hidden>
    Co-Authored-By: Hemanth Makkapati <email address hidden>