"swift_store_auth_address = ://:/v2.0/" in glance-api.conf

[Andreas Hasenack]

landscape reference: https://ci.lscape.net/job/landscape-system-tests/3027

ceph/swift autopilot deployment, using 16.04 charms.

A "juju run --all /bin/true" call that landscape makes to wait for relations to settle got stuck in a few units, glance/0 among them.

Inspection showed that /etc/glance/glance-api.conf had the [glance_store] section not fully rendered:
[glance_store]
filesystem_store_datadir = /var/lib/glance/images/
stores = glance.store.filesystem.Store,glance.store.http.Store,glance.store.swift.Store
default_store = swift
swift_store_auth_version = 2
swift_store_auth_address = ://:/v2.0/
swift_store_user = :
swift_store_key =
swift_store_create_container_on_put = True
swift_store_container = glance
swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 200
swift_enable_snet = False

glance/1 and glance/2 had it correct:
[glance_store]
filesystem_store_datadir = /var/lib/glance/images/
stores = glance.store.filesystem.Store,glance.store.http.Store,glance.store.swift.Store
default_store = swift
swift_store_auth_version = 2
swift_store_auth_address = http://10.96.16.243:5000/v2.0/
swift_store_user = services:glance
swift_store_key = mTJTkHG9zkFJbgsrtWMpLfdNjWFVKks8WYMcFqFhNLgGrRg5w2h43jZJh3rzg2Zq
swift_store_create_container_on_put = True
swift_store_container = glance
swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 200
swift_enable_snet = False

Logs and config files for glance/{0,1,2} attached. We also have the same for all the other units if needed, but it's a 89Mb zip file.

[James Page (james-page)]

"A "juju run --all /bin/true" call that landscape makes to wait for relations to settle got stuck in a few units, glance/0 among them."

would indicate that the glance/0 hook execution had not completed right? in which case I might expect that section to be incomplete

[James Page (james-page)]

The template should probably be better:

{% if swift_store -%}
swift_store_auth_version = 2
swift_store_auth_address = {{ service_protocol }}://{{ service_host }}:{{ service_port }}/v2.0/
swift_store_user = {{ admin_tenant_name }}:{{ admin_user }}
swift_store_key = {{ admin_password }}
swift_store_create_container_on_put = True
swift_store_container = glance
swift_store_large_object_size = 5120
swift_store_large_object_chunk_size = 200
swift_enable_snet = False
{% endif -%}

as it relies on keystone data, we might make that conditional on service_host being present as well before writing things out (but that's a low priority change IMHO).

[James Page (james-page)]

glance/0 was completing a db sync when the log dump was taken:

2016-07-04 00:23:27 INFO juju-log shared-db:37: Cluster leader, performing db sync
2016-07-04 00:23:28 INFO shared-db-relation-changed Option "verbose" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
2016-07-04 00:23:28 INFO shared-db-relation-changed /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py:1056: OsloDBDeprecationWarning: EngineFacade is deprecated; please use oslo_db.sqlalchemy.enginefacade
2016-07-04 00:23:28 INFO shared-db-relation-changed expire_on_commit=expire_on_commit, _conf=conf)

It looks like that was blocking - maybe that was the case for other services as well which might indicate some sort of underlying DB issue.

[James Page (james-page)]

I can't see into this problem any further given the provided data; Andreas did you figure out why hooks where not completing? and have you seen this issue since?

[Andreas Hasenack (ahasenack)]

Not exactly. I have another failed run from yesterday which has the incomplete URL in glance-api.conf, but what made the run fail was something different and it just so happens that at that point in time the rendering of glance-api.conf was incomplete in the same way.