Deleting p_s endpoint failed, it's not recreated

[Andreas Hasenack]

gs3 tries to update the product-streams by deleting and recreating it. Listing worked, deleting failed and it never got to the create part. In the end, p_s was broken in keystone catalog and juju couldn't bootstrap this cloud.

From update_product_streams_service():
        status_set('maintenance', 'Updating product-streams endpoint')

        log.info("Deleting existing product-streams endpoint: ")
        ksc.endpoints.delete(ps_endpoints[0]['id']) <----- raised an exception
        log.info("creating product-streams endpoint: {}".format(create_args))
        ksc.endpoints.create(**create_args)

The endpoint is never recreated:
$ grep "Deleting existing product-streams endpoint" glance-simplestreams-sync.log -c
2
$ grep "creating product-streams endpoint" glance-simplestreams-sync.log -c
0

The logs show that listing endpoints worked, but deleting one of the listed endpoints just stalled for 30s and then failed (full logs attached, here is just an excerpt):

Listing endpoints is OK:
DEBUG * 11-17 08:46:20 [PID:13830] * requests.packages.urllib3.connectionpool * "GET /v2.0/OS-KSADM/services HTTP/1.1" 200 1451
DEBUG * 11-17 08:46:20 [PID:13830] * keystoneclient.session * RESP: [200] Content-Length: 1451 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.1
RESP BODY: {"OS-KSADM:services": [{"id": "0fcb95a0d222422ab78fe1c4032c8b7a", "enabled": true, "type": "object-store", "name": "swift", "description": "Swift Object Stora

INFO * 11-17 08:46:20 [PID:13830] * root * ps_service_exists=True, charm_conf['use_swift']=True, swift_exists=True
...

Deleting fails:
INFO * 11-17 08:46:20 [PID:13830] * root * Updating product streams service.
DEBUG * 11-17 08:46:20 [PID:13830] * keystoneclient.session * REQ: curl -g -i -X GET http://10.96.12.80:35357/v2.0/endpoints -H "User-Agent: python-keystoneclient" -
INFO * 11-17 08:46:20 [PID:13830] * requests.packages.urllib3.connectionpool * Starting new HTTP connection (1): 10.96.12.80
DEBUG * 11-17 08:46:22 [PID:13830] * requests.packages.urllib3.connectionpool * "GET /v2.0/endpoints HTTP/1.1" 200 2685
DEBUG * 11-17 08:46:22 [PID:13830] * keystoneclient.session * RESP: [200] Content-Length: 2685 Vary: X-Auth-Token Keep-Alive: timeout=5, max=100 Server: Apache/2.4.1
RESP BODY: {"endpoints": [{"internalurl": "http://10.96.13.230:8774/v2/$(tenant_id)s", "adminurl": "http://10.96.13.230:8774/v2/$(tenant_id)s", "service_id": "7cbed4a0e2

INFO * 11-17 08:46:22 [PID:13830] * root * Executing command: ['juju-run', 'glance-simplestreams-sync/0', 'status-set maintenance "Updating product-streams endpoint
INFO * 11-17 08:46:22 [PID:13830] * root * Deleting existing product-streams endpoint:
DEBUG * 11-17 08:46:22 [PID:13830] * keystoneclient.session * REQ: curl -g -i -X DELETE http://10.96.12.80:35357/v2.0/endpoints/27386c0037fc41b18aba6b09dbe3fb21 -H "
INFO * 11-17 08:46:22 [PID:13830] * requests.packages.urllib3.connectionpool * Starting new HTTP connection (1): 10.96.12.80
ERROR * 11-17 08:46:52 [PID:13830] * root * Exception during syncing:
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 432, in main
    update_product_streams_service(ksc, services, charm_conf['region'])
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 269, in update_product_streams_service
    ksc.endpoints.delete(ps_endpoints[0]['id'])
  File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/endpoints.py", line 46, in delete
    return self._delete('/endpoints/%s' % id)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 216, in _delete
    return self.client.delete(url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 188, in delete
    return self.request(url, 'DELETE', **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 405, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 449, in _send_request
    raise exceptions.ConnectionRefused(msg)
ConnectFailure: Unable to establish connection to http://10.96.12.80:35357/v2.0/endpoints/27386c0037fc41b18aba6b09dbe3fb21

The server, however, didn't log a failure for the DELETE operation:
(...)
127.0.0.1 - - [17/Nov/2016:08:46:54 +0000] "GET /v2.0/OS-KSADM/services HTTP/1.1" 200 1759 "-" "python-keystoneclient"
127.0.0.1 - - [17/Nov/2016:08:46:54 +0000] "GET /v2.0/endpoints HTTP/1.1" 200 2825 "-" "python-keystoneclient"
10.96.10.116 - - [17/Nov/2016:08:46:22 +0000] "DELETE /v2.0/endpoints/27386c0037fc41b18aba6b09dbe3fb21 HTTP/1.1" 204 0 "-" "python-keystoneclient"
127.0.0.1 - - [17/Nov/2016:08:46:54 +0000] "GET /v2.0/users HTTP/1.1" 200 2133 "-" "python-keystoneclient"
(...)

Eventually we reach:WARNING * 11-17 08:48:19 [PID:14086] * root * found 0 product-streams endpoints in region region1, expecting one - not updating endpoint

And we have a service with no endpoint. Unfortunately I don't have a keystone catalog output for this situation.

[Andreas Hasenack (ahasenack)]

Tarball with logs from keystone-*, glance-*, swift-*.