ec2-api-metadata memcached overlap with nova-api

[Stefan Nica]

Configuring a memcached cache back-end for the ec2-api metadata service results in the following nova-api errors being reported while accessing the metadata from inside a VM:

Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/eventlet/wsgi.py", line 481, in handle_one_response
result = self.application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 216, in _call_
return app(environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 131, in _call_
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in _call_
response = req.get_response(self.application)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1316, in send
application, catch_exc_info=False)
File "/usr/lib/python2.7/site-packages/webob/request.py", line 1280, in call_application
app_iter = application(self.environ, start_response)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 131, in _call_
resp = self.call_func(req, *args, **self.kwargs)
File "/usr/lib/python2.7/site-packages/webob/dec.py", line 196, in call_func
return self.func(req, *args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 103, in _call_
meta_data = self._handle_instance_id_request(req)
File "/usr/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 178, in _handle_instance_id_request
remote_address)
File "/usr/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 297, in _get_meta_by_instance_id
elif meta_data.instance.project_id != tenant_id:
AttributeError: 'dict' object has no attribute 'instance'

[Stefan Nica (stefan.nica)]

Root cause identified: if ec2-api-metadata and nova-api are using the same memcached server, then the metadata entries will get mixed up, since the same cache entry keys are used by both.
Proposed solution: change key prefix of ec2-api entries to avoid overlap.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/504399
Committed: https://git.openstack.org/cgit/openstack/ec2-api/commit/?id=1dad720ef588fcdcf7eec409dbc472ba52f085d0
Submitter: Jenkins
Branch: master

commit 1dad720ef588fcdcf7eec409dbc472ba52f085d0
Author: Stefan Nica <email address hidden>
Date: Fri Sep 15 13:39:37 2017 +0200

    fix ec2-api-metadata memcached overlap with nova-api

    When ec2-api-metadata and nova-api are using the same
    memcached server as a cache backend, the cached values
    get intermixed due to the fact that the cache keys are
    the same. The result is that either nova-api or
    ec2-api-metadata will report cache value errors.

    This commit fixes this potential problem by using
    a different cache key prefix for the ec2-api metadata
    entries.

    Change-Id: Ibd26b33b3370d5c9c6cca0bb6e0d9c9eb03c0d50
    Closes-Bug: #1717494

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/506315

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/506315
Committed: https://git.openstack.org/cgit/openstack/ec2-api/commit/?id=1bf75237dda3d3fb5d49613c98fa5a8fbda5e098
Submitter: Jenkins
Branch: stable/pike

commit 1bf75237dda3d3fb5d49613c98fa5a8fbda5e098
Author: Stefan Nica <email address hidden>
Date: Fri Sep 15 13:39:37 2017 +0200

    fix ec2-api-metadata memcached overlap with nova-api

    When ec2-api-metadata and nova-api are using the same
    memcached server as a cache backend, the cached values
    get intermixed due to the fact that the cache keys are
    the same. The result is that either nova-api or
    ec2-api-metadata will report cache value errors.

    This commit fixes this potential problem by using
    a different cache key prefix for the ec2-api metadata
    entries.

    Change-Id: Ibd26b33b3370d5c9c6cca0bb6e0d9c9eb03c0d50
    Closes-Bug: #1717494
    (cherry picked from commit 1dad720ef588fcdcf7eec409dbc472ba52f085d0)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ec2-api 7.0.0 release.