Description ===========
Launching an instance causes a lot of error messages in nova-api logs. The instance is not able to retrieve metadata.
How to reproduce ================
Deploy nova master branch. Spawn an instance. Wait the instance to be active. In nova-api logs will see error messages.
Expected results ================
Instance retrieve metadata information
Actual results ==============
Instance is not able to retrieve metadata
Environment configuration =========================
OpenStack deployed with kolla Only source images from master fail, binary(rdo or ubuntu packages) works for now Affected CentOS, Ubuntu and OracleLinux distributions
Logs ====
All logs can be found at kolla gates:
Nova: http://logs.openstack.org/73/469373/1/check/gate-kolla-ansible-dsvm-deploy-centos-source-centos-7-nv/8cecb36/logs/kolla/nova/
Neutron: http://logs.openstack.org/73/469373/1/check/gate-kolla-ansible-dsvm-deploy-centos-source-centos-7-nv/8cecb36/logs/kolla/neutron/
Related errors:
Nova API:
2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler [req-3daa8e91-93e5-4676-b77a-048ad3dd53d2 - - - - -] Failed to get metadata for instance id: 8cbd067f-8cd6-4365-b299-3ffc146d0790 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler Traceback (most recent call last): 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 285, in _get_meta_by_instance_id 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler remote_address) 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/api/metadata/handler.py", line 87, in get_metadata_by_instance_id 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler self._cache.set(cache_key, data) 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/cache_utils.py", line 116, in set 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler return self.region.set(key, value) 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler File "/var/lib/kolla/venv/lib/python2.7/site-packages/dogpile/cache/region.py", line 973, in set 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler self.backend.set(key, self._value(value)) 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler File "/var/lib/kolla/venv/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 178, in set 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler **self.set_arguments 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_cache/backends/memcache_pool.py", line 32, in _run_method 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler return getattr(client, __name)(*args, **kwargs) 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 740, in set 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler return self._set("set", key, val, time, min_compress_len, noreply) 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1060, in _set 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler return _unsafe_set() 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 1034, in _unsafe_set 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler store_info = self._val_to_store_info(val, min_compress_len) 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler File "/var/lib/kolla/venv/lib/python2.7/site-packages/memcache.py", line 998, in _val_to_store_info 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler pickler.dump(val) 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler PicklingError: Can't pickle <class 'sqlalchemy.orm.session.Session'>: it's not the same object as sqlalchemy.orm.session.Session 2017-05-31 09:09:34.703 31 ERROR nova.api.metadata.handler
Neutron metadata:
2017-05-31 09:09:24.985 23 DEBUG neutron.agent.metadata.agent [-] Request: GET /2009-04-04/meta-data/instance-id HTTP/1.0 Accept: */* Connection: close Content-Type: text/plain Host: 169.254.169.254 User-Agent: curl/7.24.0 (x86_64-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0j zlib/1.2.6 X-Forwarded-For: 10.0.0.5 X-Neutron-Network-Id: 50395d7c-fcc6-4aef-afdc-931c4573b0d1 __call__ /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/metadata/agent.py:86 2017-05-31 09:09:25.269 23 WARNING neutron.agent.metadata.agent [-] Remote metadata server experienced an internal server error. 2017-05-31 09:09:25.269 23 INFO eventlet.wsgi.server [-] 10.0.0.5,<local> "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500 len: 361 time: 0.2839720 2017-05-31 09:09:27.460 23 DEBUG eventlet.wsgi.server [-] (23) accepted '' server /var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/wsgi.py:883
Let me know if need more information.
Regards, Eduardo
Description
===========
Launching an instance causes a lot of error messages in nova-api logs.
The instance is not able to retrieve metadata.
How to reproduce
================
Deploy nova master branch.
Spawn an instance.
Wait the instance to be active.
In nova-api logs will see error messages.
Expected results
================
Instance retrieve metadata information
Actual results
==============
Instance is not able to retrieve metadata
Environment configuration ======= ======= ====
=======
OpenStack deployed with kolla
Only source images from master fail, binary(rdo or ubuntu packages) works for now
Affected CentOS, Ubuntu and OracleLinux distributions
Logs
====
All logs can be found at kolla gates:
Nova: http:// logs.openstack. org/73/ 469373/ 1/check/ gate-kolla- ansible- dsvm-deploy- centos- source- centos- 7-nv/8cecb36/ logs/kolla/ nova/
Neutron: http:// logs.openstack. org/73/ 469373/ 1/check/ gate-kolla- ansible- dsvm-deploy- centos- source- centos- 7-nv/8cecb36/ logs/kolla/ neutron/
Related errors:
Nova API:
2017-05-31 09:09:34.703 31 ERROR nova.api. metadata. handler [req-3daa8e91- 93e5-4676- b77a-048ad3dd53 d2 - - - - -] Failed to get metadata for instance id: 8cbd067f- 8cd6-4365- b299-3ffc146d07 90 metadata. handler Traceback (most recent call last): metadata. handler File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ nova/api/ metadata/ handler. py", line 285, in _get_meta_ by_instance_ id metadata. handler remote_address) metadata. handler File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ nova/api/ metadata/ handler. py", line 87, in get_metadata_ by_instance_ id metadata. handler self._cache. set(cache_ key, data) metadata. handler File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ nova/cache_ utils.py" , line 116, in set metadata. handler return self.region. set(key, value) metadata. handler File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ dogpile/ cache/region. py", line 973, in set metadata. handler self.backend. set(key, self._value(value)) metadata. handler File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ dogpile/ cache/backends/ memcached. py", line 178, in set metadata. handler **self. set_arguments metadata. handler File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ oslo_cache/ backends/ memcache_ pool.py" , line 32, in _run_method metadata. handler return getattr(client, __name)(*args, **kwargs) metadata. handler File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ memcache. py", line 740, in set metadata. handler return self._set("set", key, val, time, min_compress_len, noreply) metadata. handler File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ memcache. py", line 1060, in _set metadata. handler return _unsafe_set() metadata. handler File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ memcache. py", line 1034, in _unsafe_set metadata. handler store_info = self._val_ to_store_ info(val, min_compress_len) metadata. handler File "/var/lib/ kolla/venv/ lib/python2. 7/site- packages/ memcache. py", line 998, in _val_to_store_info metadata. handler pickler.dump(val) metadata. handler PicklingError: Can't pickle <class 'sqlalchemy. orm.session. Session' >: it's not the same object as sqlalchemy. orm.session. Session metadata. handler
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
2017-05-31 09:09:34.703 31 ERROR nova.api.
Neutron metadata:
2017-05-31 09:09:24.985 23 DEBUG neutron. agent.metadata. agent [-] Request: GET /2009-04- 04/meta- data/instance- id HTTP/1.0 pc-linux- gnu) libcurl/7.24.0 OpenSSL/1.0.0j zlib/1.2.6 Network- Id: 50395d7c- fcc6-4aef- afdc-931c4573b0 d1 __call__ /var/lib/ kolla/venv/ lib/python2. 7/site- packages/ neutron/ agent/metadata/ agent.py: 86 agent.metadata. agent [-] Remote metadata server experienced an internal server error. wsgi.server [-] 10.0.0.5,<local> "GET /2009-04- 04/meta- data/instance- id HTTP/1.1" status: 500 len: 361 time: 0.2839720 wsgi.server [-] (23) accepted '' server /var/lib/ kolla/venv/ lib/python2. 7/site- packages/ eventlet/ wsgi.py: 883
Accept: */*
Connection: close
Content-Type: text/plain
Host: 169.254.169.254
User-Agent: curl/7.24.0 (x86_64-
X-Forwarded-For: 10.0.0.5
X-Neutron-
2017-05-31 09:09:25.269 23 WARNING neutron.
2017-05-31 09:09:25.269 23 INFO eventlet.
2017-05-31 09:09:27.460 23 DEBUG eventlet.
Let me know if need more information.
Regards, Eduardo