Nova is not actually disabling greendns

[Artom Lifshitz]

Description
===========

In [1], we began disabling greendns in eventlet to fix bug 1164822. This was done by setting the EVENTLET_NO_GREENDNS environment variable before importing eventlet. At import time, eventlet uses this env variable to enable/disable greendns [2]. Therefore, EVENTLET_NO_GREENDNS needs to be set before importing eventlet. Patch [3] changed that, setting the env var *after* importing eventlet, and thus re-enabling greendns in Nova.

Steps to reproduce
==================

-------------
Demonstration
-------------

This is a bit of a hard one to reproduce, but there's a simple way to observe the necessity of setting the env var before importing eventlet:

*** Setting the environment variable BEFORE the import ***

--- Test script ---

[artom@zoe scratchpad]$ cat eventlet-test.py
import os
import socket

os.environ['EVENTLET_NO_GREENDNS'] = 'yes'
import eventlet

eventlet.monkey_patch()
socket.gethostbyname('fake.local')

--- Result: traceback DOES NOT include greendns.py (ie, environment variable worked) ---

[artom@zoe scratchpad]$ python eventlet-test.py
Traceback (most recent call last):
  File "eventlet-test.py", line 8, in <module>
    socket.gethostbyname('fake.local')
socket.gaierror: [Errno -2] Name or service not known

*** Setting the environment variable AFTER the import ***

--- Test script ---

[artom@zoe scratchpad]$ cat eventlet-test.py
import os
import socket

import eventlet
os.environ['EVENTLET_NO_GREENDNS'] = 'yes'

eventlet.monkey_patch()
socket.gethostbyname('fake.local')

--- Result: traceback DOES include greendns.py (ie, environment variable DID NOT work) ---

[artom@zoe scratchpad]$ python eventlet-test.py
Traceback (most recent call last):
  File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 424, in resolve
    return _proxy.query(name, rdtype, raise_on_no_answer=raises,
  File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 380, in query
    return end()
  File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 359, in end
    raise result[1]
  File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 340, in step
    a = fun(*args, **kwargs)
  File "/home/artom/.local/lib/python3.8/site-packages/dns/resolver.py", line 1002, in query
    raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)
dns.resolver.NXDOMAIN: None of DNS query names exist: fake.local., fake.local.redhat.com.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "eventlet-test.py", line 8, in <module>
    socket.gethostbyname('fake.local')
  File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 550, in gethostbyname
    rrset = resolve(hostname)
  File "/home/artom/.local/lib/python3.8/site-packages/eventlet/support/greendns.py", line 434, in resolve
    raise EAI_NODATA_ERROR
socket.gaierror: [Errno -2] Name or service not known

----------------
Real life impact
----------------

Downstream in our openstack product, the return of eventlet has been reported [4] as the cause of needless delays when failing over between Rabbit URLs in case one of the Rabbit servers goes down. In our deployment, DNS is not used, and IPv4 hostnames are just written to /etc/hosts. Looks like greendns tries to to IPv6 resolution regardless, needlessly using up 30 seconds in order to time out.

Expected result
===============

Almost-immediate failover to new Rabbit server.

Actual result
=============

greendns attempts IPv6 name resolution (though unclear of which Rabbit server - the failed one, or the next one), before finally, after 30 seconds, connecting to the next Rabbit server.

Environment
===========

This has been reported on stable/train, but should be the same on master.

References
==========

[1] https://review.opendev.org/#/c/26325/
[2] https://github.com/eventlet/eventlet/blob/v0.26.0/eventlet/green/socket.py#L20
[3] https://review.opendev.org/#/c/626952/
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1860818

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/751302
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7c1d964faab33a02fe2366b5194611252be045fc
Submitter: Zuul
Branch: master

commit 7c1d964faab33a02fe2366b5194611252be045fc
Author: Artom Lifshitz <email address hidden>
Date: Fri Sep 11 10:23:30 2020 -0400

    Correctly disable greendns

    Previously, we were setting the environment variable to disable
    greendns in eventlet *after* import eventlet. This has no effect, as
    eventlet processes environment variables at import time. This patch
    moves the setting of EVENTLET_NO_GREENDNS before importing eventlet in
    order to correctly disable greendns.

    Closes-bug: 1895322
    Change-Id: I4deed815c8984df095019a7f61d089f233f1fc66

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/751621

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/751622

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/751621
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=79e6b7fd30a04cdb2374abcaf496b6b5b76084ff
Submitter: Zuul
Branch: stable/ussuri

commit 79e6b7fd30a04cdb2374abcaf496b6b5b76084ff
Author: Artom Lifshitz <email address hidden>
Date: Fri Sep 11 10:23:30 2020 -0400

    Correctly disable greendns

    Previously, we were setting the environment variable to disable
    greendns in eventlet *after* import eventlet. This has no effect, as
    eventlet processes environment variables at import time. This patch
    moves the setting of EVENTLET_NO_GREENDNS before importing eventlet in
    order to correctly disable greendns.

    Closes-bug: 1895322
    Change-Id: I4deed815c8984df095019a7f61d089f233f1fc66
    (cherry picked from commit 7c1d964faab33a02fe2366b5194611252be045fc)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/751622
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4984b3b75b5f7086b7464d9acb940e3fb0334a5e
Submitter: Zuul
Branch: stable/train

commit 4984b3b75b5f7086b7464d9acb940e3fb0334a5e
Author: Artom Lifshitz <email address hidden>
Date: Fri Sep 11 10:23:30 2020 -0400

    Correctly disable greendns

    Previously, we were setting the environment variable to disable
    greendns in eventlet *after* import eventlet. This has no effect, as
    eventlet processes environment variables at import time. This patch
    moves the setting of EVENTLET_NO_GREENDNS before importing eventlet in
    order to correctly disable greendns.

    Closes-bug: 1895322
    Change-Id: I4deed815c8984df095019a7f61d089f233f1fc66
    (cherry picked from commit 7c1d964faab33a02fe2366b5194611252be045fc)
    (cherry picked from commit 79e6b7fd30a04cdb2374abcaf496b6b5b76084ff)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/753072
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6f35e4fd2afdb8e26682fc5bd73dbe23236d25c7
Submitter: Zuul
Branch: master

commit 6f35e4fd2afdb8e26682fc5bd73dbe23236d25c7
Author: Artom Lifshitz <email address hidden>
Date: Mon Sep 21 11:47:08 2020 -0400

    Test for disabling greendns

    In commit 7c1d964faa we fixed how we disable greendns. This patch adds
    a test for this. It also lays down the groundwork for future tests
    of how we manage eventlet's monkeypatching.

    How and what eventlet monkeypatches can be controlled by environment
    variables that are processed by eventlet at import-time (for exmaple,
    EVENTLET_NO_GREENDNS). Nova manages all of this in nova.monkey_patch.
    Therefore, nova.monkey_patch must be the first thing to import
    eventlet. As nova.tests.functional.__init__ imports nova.monkey_patch,
    our new test can go in the functional tree.

    Related-bug: 1895322
    Change-Id: I5b6c45b7b9a9eca3c13ecfaa5f50942922b69270

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/754220

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/754220
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9ac794b93b827dde1d9545b88fd7913b7c00dacb
Submitter: Zuul
Branch: stable/ussuri

commit 9ac794b93b827dde1d9545b88fd7913b7c00dacb
Author: Artom Lifshitz <email address hidden>
Date: Mon Sep 21 11:47:08 2020 -0400

    Test for disabling greendns

    In commit 7c1d964faa we fixed how we disable greendns. This patch adds
    a test for this. It also lays down the groundwork for future tests
    of how we manage eventlet's monkeypatching.

    How and what eventlet monkeypatches can be controlled by environment
    variables that are processed by eventlet at import-time (for exmaple,
    EVENTLET_NO_GREENDNS). Nova manages all of this in nova.monkey_patch.
    Therefore, nova.monkey_patch must be the first thing to import
    eventlet. As nova.tests.functional.__init__ imports nova.monkey_patch,
    our new test can go in the functional tree.

    Related-bug: 1895322
    Change-Id: I5b6c45b7b9a9eca3c13ecfaa5f50942922b69270
    (cherry picked from commit 6f35e4fd2afdb8e26682fc5bd73dbe23236d25c7)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/761763

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova stein-eol release.