Glance api service crashes but restarts infinitely
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Glance |
New
|
Undecided
|
Unassigned |
Bug Description
Openstack release in which defect is being reported for: Mitaka
OS: CentOS7
Component: Glance
Specific: Glance-api
# rpm -qa | grep glance
python-
python-
python-
openstack-
Glance store settings in /etc/glance/
[glance_store]
# From glance.store
stores = vmware
default_store = vmware
vmware_server_host = 10.10.1.10
vmware_
vmware_
vmware_
vmware_insecure = true
vmware_datastores = DC:DS01:200
vmware_datastores = DC:DS02:100
/var/log/
2016-09-13 17:50:08.636 27351 WARNING glance_
2016-09-13 17:52:08.779 27526 INFO oslo_vmware.api [-] Successfully established new session; session ID is 6d8cf.
2016-09-13 17:52:16.370 27552 INFO oslo_vmware.api [-] Successfully established new session; session ID is e4c65.
2016-09-13 17:52:24.118 27566 INFO oslo_vmware.api [-] Successfully established new session; session ID is 41c30.
2016-09-13 17:52:31.947 27580 INFO oslo_vmware.api [-] Successfully established new session; session ID is 80b67.
2016-09-13 17:52:39.598 27592 INFO oslo_vmware.api [-] Successfully established new session; session ID is c5356.
2016-09-13 17:52:47.107 27630 INFO oslo_vmware.api [-] Successfully established new session; session ID is 5d183.
2016-09-13 17:52:54.912 27644 INFO oslo_vmware.api [-] Successfully established new session; session ID is d0a10.
2016-09-13 17:53:02.620 27658 INFO oslo_vmware.api [-] Successfully established new session; session ID is f4f7f.
2016-09-13 17:53:10.404 27670 INFO oslo_vmware.api [-] Successfully established new session; session ID is 9ba5a.
2016-09-13 17:53:18.106 27708 INFO oslo_vmware.api [-] Successfully established new session; session ID is 46e93.
2016-09-13 17:53:25.602 27725 INFO oslo_vmware.api [-] Successfully established new session; session ID is 6bad2.
2016-09-13 17:53:33.135 27748 INFO oslo_vmware.api [-] Successfully established new session; session ID is 81b90.
2016-09-13 17:53:40.905 27760 INFO oslo_vmware.api [-] Successfully established new session; session ID is 04754.
Two observations from above log:
1. After changing the default_store and stores value to 'vmware' from 'vsphere', started seeing the 'Successfully established new session' messages in api.log
2. Tried investigating as to why there are multiple 'new session' in api.log. After looking into /var/log/messages witnessed the below mentioned log:
Sep 13 18:09:06 controller01 glance-api: /usr/lib/
Sep 13 18:09:06 controller01 glance-api: InsecureRequest
Sep 13 18:09:06 controller01 glance-api: ERROR: Store for scheme vmware not found
Sep 13 18:09:06 controller01 systemd: openstack-
Sep 13 18:09:06 controller01 systemd: Unit openstack-
Sep 13 18:09:06 controller01 systemd: openstack-
Sep 13 18:09:06 controller01 systemd: openstack-
Sep 13 18:09:06 controller01 systemd: Started OpenStack Image Service (code-named Glance) API server.
Sep 13 18:09:06 controller01 systemd: Starting OpenStack Image Service (code-named Glance) API server...
Sep 13 18:09:06 controller01 glance-api: /usr/lib/
1. Remove/comment stores and retain only default_store with the value 'vmware', glance api service would crash instantly and stay in that state forever.
2. Retain both stores and default_store with the value 'vsphere', glance api service would crash instantly and stay in that state forever
3. Remove/comment stores and retain only default_store with the value 'vsphere', glance api service would crash instantly and stay in that state forever
4. Retain both stores and default_store with the value 'vmware', glance api service would crash in about 8 seconds and restart automatically, but would loop between crash and restart forever, which is the reason we keep seeing, multiple new session established messages in api.log
Backend vSphere is v5.5U2
Please let know if this is known issue in Mitaka or there needs some config changes to be done from my end
Hello,
I have also this issue, and I believe the problem is in these parts of the code:
glance_ store/backend. py:
def verify_ default_ store() : store.default_ store
scheme_name = CONF.glance_
...
def add_to_ backend( conf, image_id, data, size, scheme=None, context=None,
verifier= None): store'] ['default_ store']
if scheme is None:
scheme = conf['glance_
Both seem to assume that there's an identity between store name and scheme, which is no longer true for vmware datastore.
In fact datastore name is expected to be vmware while SCHEME is vsphere.
To workaround this I've written (based on get_store_ from_scheme) a new function that get datastore form its name.
+def get_store_ from_name( store_name) : SCHEME_ TO_CLS_ MAP.items( ): 'store_ entry'] == store_name: 'store' ] capable( capabilities. BitMasks. DRIVER_ REUSABLE) : 'store_ entry'] store.conf, store_entry, invoke_load=True) store_location_ class() schemes( ): register_ scheme_ map(scheme_ map) rror: 'store' ] = store UnknownStore( store_name= store_name)
+ store = None
+ for scheme, store_value in location.
+ if store_value[
+ store = store_value[
+ if not store.is_
+ # Driver instance isn't stateless so it can't
+ # be reused safely and need recreation.
+ store_entry = store_value[
+ store = _load_store(
+ store.configure()
+ try:
+ scheme_map = {}
+ loc_cls = store.get_
+ for scheme in store.get_
+ scheme_map[scheme] = {
+ 'store': store,
+ 'location_class': loc_cls,
+ 'store_entry': store_entry
+ }
+ location.
+ except NotImplementedE
+ store_value[
+ if store is None:
+ raise exceptions.
+ return store
and changed the above functions to use this new one:
def add_to_ backend( conf, image_id, data, size, scheme=None, context=None,
verifier= None): store'] ['default_ store'] from_name( store_name) from_scheme( scheme) to_backend( image_id, data, size, store, context,
verifier)
if scheme is None:
store_name = conf['glance_
store = get_store_
else:
store = get_store_
return store_add_
def verify_ default_ store() : store.default_ store
get_store_ from_name( store_name) UnknownStore:
store_name = CONF.glance_
try:
except exceptions.
msg = _("Store named %s not found") % store_name
raise RuntimeError(msg)
glance_ store/exception s.py:
+class UnknownStore( GlanceStoreExce ption):
+ message = _("Unknown scheme '%(scheme)s' found in URI")
This way everything works, at least for me.
Hope it helps.