[FT] neutron.tests.functional.agent.linux.test_iptables.IptablesManagerNonRootTestCase test cases always failing

Bug #1888213 reported by Rodolfo Alonso on 2020-07-20
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
High
Rodolfo Alonso

Bug Description

Maybe (just a guess) [1] could be the culprit. Still investigating...

[1]https://github.com/openstack/stevedore/commit/d5297167e08468c75d2477f15004df61cf98e57e

Changed in neutron:
importance: Undecided → High
description: updated

Ok, this is the error:

>>> import importlib.metadata
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'importlib.metadata'
>>>

But [1] is only catching ImportError [2]

[2]https://github.com/openstack/stevedore/commit/d5297167e08468c75d2477f15004df61cf98e57e#diff-11dca3f8d632f130fa82203beb6dd8cdR29

no longer affects: stevedore

ModuleNotFoundError is a child class of ImportError, that should be fine then.

no longer affects: python-stevedore

I think we have a circular reference. See http://paste.openstack.org/show/796122/. That's happening since I8b08f289d446f4775eac1e1a91997fa96f25f641.

oslo_config.cfg calls stevedore
--> calls importlib_metadata
--> imports oslo_log.log
--> imports oslo_log.versionutils
--> imports oslo_config.cfg !!

Two items there:
- remove oslo_service log import (not needed).
- configure logging before dynamically loading any module

Fix proposed to branch: master
Review: https://review.opendev.org/741957

Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
status: New → In Progress

Hello:

Same problem is happening in fullstack tests. E.g.: https://eb61bdf941c65af3de29-55f728dd0276ae72b3fe1bf93bf80aa1.ssl.cf2.rackcdn.com/740949/1/check/neutron-fullstack/5d48dbd/testr_results.html

From a fullstack test log: http://paste.openstack.org/show/796132/

The oslo_config.cfg module should be loaded before eventlet.

tags: added: gate-failure

Reviewed: https://review.opendev.org/741957
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bc99e56515537210a35988d3fbdec0669469d0b3
Submitter: Zuul
Branch: master

commit bc99e56515537210a35988d3fbdec0669469d0b3
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 20 14:12:06 2020 +0000

    Import "oslo_config.cfg" before "eventlet"

    Eventlet will patch the "os" library and how it works.

    If "oslo_config.cfg" is called after "eventlet":
    - "oslo_config.cfg" imports "stevedore"
    - "stevedore" imports "importlib_metadata"
    - this library will try to use "os.open" library, but patched
    - "eventlet" imports "oslo_service"
    - "oslo_service" imports "oslo_logging"
    - "oslo_logging" imports "oslo_config.cfg", but at this point,
      the instance cfg.CONF does not exist yet

    If "oslo_config.cfg" is imported before calling eventlet,
    "oslo_logging" is not called and there is no circular import.

    Closes-Bug: #1888213

    [1]I8b08f289d446f4775eac1e1a91997fa96f25f641

    Change-Id: Idcc04f46b2a01e24dfd56cef8121694439e0339e

Changed in neutron:
status: In Progress → Fix Released
tags: added: neutron-proactive-backport-potential

Reviewed: https://review.opendev.org/757059
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=389bae74aedd7c58e61d9ad13def59cbf098e00a
Submitter: Zuul
Branch: stable/ussuri

commit 389bae74aedd7c58e61d9ad13def59cbf098e00a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 20 14:12:06 2020 +0000

    Import "oslo_config.cfg" before "eventlet"

    Eventlet will patch the "os" library and how it works.

    If "oslo_config.cfg" is called after "eventlet":
    - "oslo_config.cfg" imports "stevedore"
    - "stevedore" imports "importlib_metadata"
    - this library will try to use "os.open" library, but patched
    - "eventlet" imports "oslo_service"
    - "oslo_service" imports "oslo_logging"
    - "oslo_logging" imports "oslo_config.cfg", but at this point,
      the instance cfg.CONF does not exist yet

    If "oslo_config.cfg" is imported before calling eventlet,
    "oslo_logging" is not called and there is no circular import.

    Closes-Bug: #1888213

    [1]I8b08f289d446f4775eac1e1a91997fa96f25f641

    Change-Id: Idcc04f46b2a01e24dfd56cef8121694439e0339e
    (cherry picked from commit bc99e56515537210a35988d3fbdec0669469d0b3)

tags: added: in-stable-ussuri

Reviewed: https://review.opendev.org/757060
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a9af91846a1e6ff2389437eb8cf87fd1810fe35d
Submitter: Zuul
Branch: stable/train

commit a9af91846a1e6ff2389437eb8cf87fd1810fe35d
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 20 14:12:06 2020 +0000

    Import "oslo_config.cfg" before "eventlet"

    Eventlet will patch the "os" library and how it works.

    If "oslo_config.cfg" is called after "eventlet":
    - "oslo_config.cfg" imports "stevedore"
    - "stevedore" imports "importlib_metadata"
    - this library will try to use "os.open" library, but patched
    - "eventlet" imports "oslo_service"
    - "oslo_service" imports "oslo_logging"
    - "oslo_logging" imports "oslo_config.cfg", but at this point,
      the instance cfg.CONF does not exist yet

    If "oslo_config.cfg" is imported before calling eventlet,
    "oslo_logging" is not called and there is no circular import.

    Closes-Bug: #1888213

    [1]I8b08f289d446f4775eac1e1a91997fa96f25f641

    Change-Id: Idcc04f46b2a01e24dfd56cef8121694439e0339e
    (cherry picked from commit bc99e56515537210a35988d3fbdec0669469d0b3)

tags: added: in-stable-train
tags: added: in-stable-stein

Reviewed: https://review.opendev.org/757061
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=20e8c6f55380bc94709bc3f25d5c8eebd1f56bee
Submitter: Zuul
Branch: stable/stein

commit 20e8c6f55380bc94709bc3f25d5c8eebd1f56bee
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 20 14:12:06 2020 +0000

    Import "oslo_config.cfg" before "eventlet"

    Eventlet will patch the "os" library and how it works.

    If "oslo_config.cfg" is called after "eventlet":
    - "oslo_config.cfg" imports "stevedore"
    - "stevedore" imports "importlib_metadata"
    - this library will try to use "os.open" library, but patched
    - "eventlet" imports "oslo_service"
    - "oslo_service" imports "oslo_logging"
    - "oslo_logging" imports "oslo_config.cfg", but at this point,
      the instance cfg.CONF does not exist yet

    If "oslo_config.cfg" is imported before calling eventlet,
    "oslo_logging" is not called and there is no circular import.

    Closes-Bug: #1888213

    [1]I8b08f289d446f4775eac1e1a91997fa96f25f641

    Change-Id: Idcc04f46b2a01e24dfd56cef8121694439e0339e
    (cherry picked from commit bc99e56515537210a35988d3fbdec0669469d0b3)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers