neutron

[FT] neutron.tests.functional.agent.linux.test_iptables.IptablesManagerNonRootTestCase test cases always failing

Bug #1888213 reported by Rodolfo Alonso
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Rodolfo Alonso

Bug Description

Test cases failing:
- test_binary_name
- test_binary_name_eventlet_spawn

Those tests are blocking the CI.

Logs: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9c1/735802/3/check/neutron-functional/9c1f55b/testr_results.html

Snippet: http://paste.openstack.org/show/796114/

See original description
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Maybe (just a guess) [1] could be the culprit. Still investigating...

[1]https://github.com/openstack/stevedore/commit/d5297167e08468c75d2477f15004df61cf98e57e

Changed in neutron:
importance: Undecided → High
description: updated
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Ok, this is the error:

>>> import importlib.metadata
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'importlib.metadata'
>>>

But [1] is only catching ImportError [2]

[2]https://github.com/openstack/stevedore/commit/d5297167e08468c75d2477f15004df61cf98e57e#diff-11dca3f8d632f130fa82203beb6dd8cdR29

no longer affects: stevedore
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

ModuleNotFoundError is a child class of ImportError, that should be fine then.

no longer affects: python-stevedore
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

I think we have a circular reference. See http://paste.openstack.org/show/796122/. That's happening since I8b08f289d446f4775eac1e1a91997fa96f25f641.

oslo_config.cfg calls stevedore
--> calls importlib_metadata
--> imports oslo_log.log
--> imports oslo_log.versionutils
--> imports oslo_config.cfg !!

Two items there:
- remove oslo_service log import (not needed).
- configure logging before dynamically loading any module

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/741957

Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
status: New → In Progress
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Hello:

Same problem is happening in fullstack tests. E.g.: https://eb61bdf941c65af3de29-55f728dd0276ae72b3fe1bf93bf80aa1.ssl.cf2.rackcdn.com/740949/1/check/neutron-fullstack/5d48dbd/testr_results.html

From a fullstack test log: http://paste.openstack.org/show/796132/

The oslo_config.cfg module should be loaded before eventlet.

Nate Johnston (nate-johnston)
tags: added: gate-failure
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/741957
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bc99e56515537210a35988d3fbdec0669469d0b3
Submitter: Zuul
Branch: master

commit bc99e56515537210a35988d3fbdec0669469d0b3
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 20 14:12:06 2020 +0000

    Import "oslo_config.cfg" before "eventlet"

    Eventlet will patch the "os" library and how it works.

    If "oslo_config.cfg" is called after "eventlet":
    - "oslo_config.cfg" imports "stevedore"
    - "stevedore" imports "importlib_metadata"
    - this library will try to use "os.open" library, but patched
    - "eventlet" imports "oslo_service"
    - "oslo_service" imports "oslo_logging"
    - "oslo_logging" imports "oslo_config.cfg", but at this point,
      the instance cfg.CONF does not exist yet

    If "oslo_config.cfg" is imported before calling eventlet,
    "oslo_logging" is not called and there is no circular import.

    Closes-Bug: #1888213

    [1]I8b08f289d446f4775eac1e1a91997fa96f25f641

    Change-Id: Idcc04f46b2a01e24dfd56cef8121694439e0339e

Changed in neutron:
status: In Progress → Fix Released
Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/757059

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/757060

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/757061

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/757062

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/757063

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/ussuri)

Reviewed: https://review.opendev.org/757059
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=389bae74aedd7c58e61d9ad13def59cbf098e00a
Submitter: Zuul
Branch: stable/ussuri

commit 389bae74aedd7c58e61d9ad13def59cbf098e00a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 20 14:12:06 2020 +0000

    Import "oslo_config.cfg" before "eventlet"

    Eventlet will patch the "os" library and how it works.

    If "oslo_config.cfg" is called after "eventlet":
    - "oslo_config.cfg" imports "stevedore"
    - "stevedore" imports "importlib_metadata"
    - this library will try to use "os.open" library, but patched
    - "eventlet" imports "oslo_service"
    - "oslo_service" imports "oslo_logging"
    - "oslo_logging" imports "oslo_config.cfg", but at this point,
      the instance cfg.CONF does not exist yet

    If "oslo_config.cfg" is imported before calling eventlet,
    "oslo_logging" is not called and there is no circular import.

    Closes-Bug: #1888213

    [1]I8b08f289d446f4775eac1e1a91997fa96f25f641

    Change-Id: Idcc04f46b2a01e24dfd56cef8121694439e0339e
    (cherry picked from commit bc99e56515537210a35988d3fbdec0669469d0b3)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/757060
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a9af91846a1e6ff2389437eb8cf87fd1810fe35d
Submitter: Zuul
Branch: stable/train

commit a9af91846a1e6ff2389437eb8cf87fd1810fe35d
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 20 14:12:06 2020 +0000

    Import "oslo_config.cfg" before "eventlet"

    Eventlet will patch the "os" library and how it works.

    If "oslo_config.cfg" is called after "eventlet":
    - "oslo_config.cfg" imports "stevedore"
    - "stevedore" imports "importlib_metadata"
    - this library will try to use "os.open" library, but patched
    - "eventlet" imports "oslo_service"
    - "oslo_service" imports "oslo_logging"
    - "oslo_logging" imports "oslo_config.cfg", but at this point,
      the instance cfg.CONF does not exist yet

    If "oslo_config.cfg" is imported before calling eventlet,
    "oslo_logging" is not called and there is no circular import.

    Closes-Bug: #1888213

    [1]I8b08f289d446f4775eac1e1a91997fa96f25f641

    Change-Id: Idcc04f46b2a01e24dfd56cef8121694439e0339e
    (cherry picked from commit bc99e56515537210a35988d3fbdec0669469d0b3)

tags: added: in-stable-train
tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/757061
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=20e8c6f55380bc94709bc3f25d5c8eebd1f56bee
Submitter: Zuul
Branch: stable/stein

commit 20e8c6f55380bc94709bc3f25d5c8eebd1f56bee
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 20 14:12:06 2020 +0000

    Import "oslo_config.cfg" before "eventlet"

    Eventlet will patch the "os" library and how it works.

    If "oslo_config.cfg" is called after "eventlet":
    - "oslo_config.cfg" imports "stevedore"
    - "stevedore" imports "importlib_metadata"
    - this library will try to use "os.open" library, but patched
    - "eventlet" imports "oslo_service"
    - "oslo_service" imports "oslo_logging"
    - "oslo_logging" imports "oslo_config.cfg", but at this point,
      the instance cfg.CONF does not exist yet

    If "oslo_config.cfg" is imported before calling eventlet,
    "oslo_logging" is not called and there is no circular import.

    Closes-Bug: #1888213

    [1]I8b08f289d446f4775eac1e1a91997fa96f25f641

    Change-Id: Idcc04f46b2a01e24dfd56cef8121694439e0339e
    (cherry picked from commit bc99e56515537210a35988d3fbdec0669469d0b3)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/757063
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=274a75497a161df15d0dafe85179b4ace3c113e7
Submitter: Zuul
Branch: stable/queens

commit 274a75497a161df15d0dafe85179b4ace3c113e7
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 20 14:12:06 2020 +0000

    Import "oslo_config.cfg" before "eventlet"

    Eventlet will patch the "os" library and how it works.

    If "oslo_config.cfg" is called after "eventlet":
    - "oslo_config.cfg" imports "stevedore"
    - "stevedore" imports "importlib_metadata"
    - this library will try to use "os.open" library, but patched
    - "eventlet" imports "oslo_service"
    - "oslo_service" imports "oslo_logging"
    - "oslo_logging" imports "oslo_config.cfg", but at this point,
      the instance cfg.CONF does not exist yet

    If "oslo_config.cfg" is imported before calling eventlet,
    "oslo_logging" is not called and there is no circular import.

    Closes-Bug: #1888213

    [1]I8b08f289d446f4775eac1e1a91997fa96f25f641

    Change-Id: Idcc04f46b2a01e24dfd56cef8121694439e0339e
    (cherry picked from commit bc99e56515537210a35988d3fbdec0669469d0b3)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/757062
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=56f2bf37d18ec1ffb9dd618d5214dfeef00da8d0
Submitter: Zuul
Branch: stable/rocky

commit 56f2bf37d18ec1ffb9dd618d5214dfeef00da8d0
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jul 20 14:12:06 2020 +0000

    Import "oslo_config.cfg" before "eventlet"

    Eventlet will patch the "os" library and how it works.

    If "oslo_config.cfg" is called after "eventlet":
    - "oslo_config.cfg" imports "stevedore"
    - "stevedore" imports "importlib_metadata"
    - this library will try to use "os.open" library, but patched
    - "eventlet" imports "oslo_service"
    - "oslo_service" imports "oslo_logging"
    - "oslo_logging" imports "oslo_config.cfg", but at this point,
      the instance cfg.CONF does not exist yet

    If "oslo_config.cfg" is imported before calling eventlet,
    "oslo_logging" is not called and there is no circular import.

    Closes-Bug: #1888213

    [1]I8b08f289d446f4775eac1e1a91997fa96f25f641

    Change-Id: Idcc04f46b2a01e24dfd56cef8121694439e0339e
    (cherry picked from commit bc99e56515537210a35988d3fbdec0669469d0b3)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 15.3.1

This issue was fixed in the openstack/neutron 15.3.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.3.0

This issue was fixed in the openstack/neutron 16.3.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron queens-eol

This issue was fixed in the openstack/neutron queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron rocky-eol

This issue was fixed in the openstack/neutron rocky-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.