504 Gateway Timeout while trying to verify_metadata for a server
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| OpenStack-Gate |
New
|
Undecided
|
Unassigned | ||
Bug Description
Seen in the gate recently where a test fails while trying to verify metadata, example:
Traceback (most recent call last):
File "/opt/stack/
return f(*func_args, **func_kwargs)
File "/opt/stack/
self.
File "/opt/stack/
if not test_utils.
File "/opt/stack/
if func(*args, **kwargs):
File "/opt/stack/
self.
File "/opt/stack/
self.
File "/opt/stack/
raise mismatch_error
testtools.
reference = '172.24.5.114'
actual = '''\
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
'''
: Failed while verifying metadata on server. Result of command "curl http://
Logstash query:
54 hits in the last 7 days, all failures
Here is what we currently use for the metadata API config nova-metadata-
[uwsgi]
http = 0.0.0.0:8775
lazy-apps = true
add-header = Connection: close
buffer-size = 65535
hook-master-start = unix_signal:15 gracefully_
thunder-lock = true
plugins = http,python3
enable-threads = true
worker-reload-mercy = 90
exit-on-reload = false
die-on-term = true
master = true
processes = 2
wsgi-file = /usr/local/
We're using 2 API workers here, as specified by the formula [1]:
# Sets the maximum number of workers for most services to reduce
# the memory used where there are a large number of CPUs present
# (the default number of workers for many services is the number of CPUs)
# Also sets the minimum number of workers to 2.
API_WORKERS=
I'm wondering if we should try increasing this for the metadata API only, considering how many gate failures we observe around metadata API timing out or otherwise failing to respond to requests [2][3].
[1] https:/
[2] https:/
[3] https:/
