Nova Meta Data service stack traces with error: [Errno 113] No route to host all the time

[Sean Dague]

The Nova Meta Data service is full of stack traces. Largely all about no route to host. These can be seen like this:

2014-04-27 19:10:41.463 30617 INFO nova.metadata.wsgi.server [-] 10.1.0.71 "GET /2009-04-04/meta-data/placement/availability-zone HTTP/1.1" status: 200 len: 119 time: 0.0008161
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl
    func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 594, in process_request
    proto.__init__(sock, address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 638, in __init__
    self.handle()
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 342, in handle
    self.handle_one_request()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 231, in handle_one_request
    self.raw_requestline = self.rfile.readline(self.server.url_length_limit)
  File "/usr/lib/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio.py", line 253, in recv
    return fd.recv(buflen, flags)
error: [Errno 113] No route to host

Example log: http://logs.openstack.org/43/90043/1/check/check-tempest-dsvm-postgres-full-icehouse/7c4098a/logs/screen-n-api-meta.txt.gz

While these are mostly non fatal, it appears sometimes these trigger a crash, causing test runs to fail.

This is happening *a lot*: message:"error: [Errno 113] No route to host" AND (tags:screen-n-api-meta.txt OR tags:screen-n-api.txt)

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiZXJyb3I6IFtFcnJubyAxMTNdIE5vIHJvdXRlIHRvIGhvc3RcIiBBTkQgKHRhZ3M6c2NyZWVuLW4tYXBpLW1ldGEudHh0IE9SIHRhZ3M6c2NyZWVuLW4tYXBpLnR4dCkiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTg2ODM5MzQyNTh9

[Michael Kazakov (gnomino)]

Do you use neutron-metadata-proxy?
Post nova.conf from controller and compute node
Show your fierwall rules from controller and compute node