Nova is not actually disabling greendns
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Undecided
|
Artom Lifshitz | ||
Train |
Fix Released
|
Undecided
|
Unassigned | ||
Ussuri |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
Description
===========
In [1], we began disabling greendns in eventlet to fix bug 1164822. This was done by setting the EVENTLET_
Steps to reproduce
==================
-------------
Demonstration
-------------
This is a bit of a hard one to reproduce, but there's a simple way to observe the necessity of setting the env var before importing eventlet:
*** Setting the environment variable BEFORE the import ***
--- Test script ---
[artom@zoe scratchpad]$ cat eventlet-test.py
import os
import socket
os.environ[
import eventlet
eventlet.
socket.
--- Result: traceback DOES NOT include greendns.py (ie, environment variable worked) ---
[artom@zoe scratchpad]$ python eventlet-test.py
Traceback (most recent call last):
File "eventlet-test.py", line 8, in <module>
socket.
socket.gaierror: [Errno -2] Name or service not known
*** Setting the environment variable AFTER the import ***
--- Test script ---
[artom@zoe scratchpad]$ cat eventlet-test.py
import os
import socket
import eventlet
os.environ[
eventlet.
socket.
--- Result: traceback DOES include greendns.py (ie, environment variable DID NOT work) ---
[artom@zoe scratchpad]$ python eventlet-test.py
Traceback (most recent call last):
File "/home/
return _proxy.query(name, rdtype, raise_on_
File "/home/
return end()
File "/home/
raise result[1]
File "/home/
a = fun(*args, **kwargs)
File "/home/
raise NXDOMAIN(
dns.resolver.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "eventlet-test.py", line 8, in <module>
socket.
File "/home/
rrset = resolve(hostname)
File "/home/
raise EAI_NODATA_ERROR
socket.gaierror: [Errno -2] Name or service not known
----------------
Real life impact
----------------
Downstream in our openstack product, the return of eventlet has been reported [4] as the cause of needless delays when failing over between Rabbit URLs in case one of the Rabbit servers goes down. In our deployment, DNS is not used, and IPv4 hostnames are just written to /etc/hosts. Looks like greendns tries to to IPv6 resolution regardless, needlessly using up 30 seconds in order to time out.
Expected result
===============
Almost-immediate failover to new Rabbit server.
Actual result
=============
greendns attempts IPv6 name resolution (though unclear of which Rabbit server - the failed one, or the next one), before finally, after 30 seconds, connecting to the next Rabbit server.
Environment
===========
This has been reported on stable/train, but should be the same on master.
References
==========
[1] https:/
[2] https:/
[3] https:/
[4] https:/
Changed in nova: | |
assignee: | nobody → Artom Lifshitz (notartom) |
status: | New → In Progress |
Reviewed: https:/ /review. opendev. org/751302 /git.openstack. org/cgit/ openstack/ nova/commit/ ?id=7c1d964faab 33a02fe2366b519 4611252be045fc
Committed: https:/
Submitter: Zuul
Branch: master
commit 7c1d964faab33a0 2fe2366b5194611 252be045fc
Author: Artom Lifshitz <email address hidden>
Date: Fri Sep 11 10:23:30 2020 -0400
Correctly disable greendns
Previously, we were setting the environment variable to disable NO_GREENDNS before importing eventlet in
greendns in eventlet *after* import eventlet. This has no effect, as
eventlet processes environment variables at import time. This patch
moves the setting of EVENTLET_
order to correctly disable greendns.
Closes-bug: 1895322 f095019a7f61d08 9f233f1fc66
Change-Id: I4deed815c8984d