[SRU] ML2 plugin fails with "AttributeError: module 'select' has no attribute 'poll'"

Bug #1889396 reported by Aurelien Lourot on 2020-07-29
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
High
Unassigned
Stein
High
Unassigned
Train
High
Unassigned
Ussuri
High
Unassigned
networking-arista
Undecided
Unassigned
networking-arista (Ubuntu)
High
Unassigned
Focal
High
Unassigned
Groovy
High
Unassigned

Bug Description

[Impact]
When running the latest Neutron ML2 Arista plugin on Ubuntu 18.04 with Python 3.6 I get:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/networking_arista/ml2/arista_sync.py", line 255, in sync_loop
    sync_required = self.wait_for_sync_required()
  File "/usr/lib/python3/dist-packages/networking_arista/ml2/arista_sync.py", line 184, in wait_for_sync_required
    return self.wait_for_mech_driver_update(timeout)
  File "/usr/lib/python3/dist-packages/networking_arista/ml2/arista_sync.py", line 165, in wait_for_mech_driver_update
    resource = self.provision_queue.get(timeout=timeout)
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 104, in get
    if not self._poll(timeout):
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 903, in wait
    with _WaitSelector() as selector:
  File "/usr/lib/python3.6/selectors.py", line 348, in __init__
    self._poll = select.poll()
AttributeError: module 'select' has no attribute 'poll'

This is because arista_sync.py imports eventlet, which removes poll() from select as it is considered "non-green", see https://github.com/eventlet/eventlet/issues/608 . However earlier in mechanism_arista.py multiprocessing was imported, which then adjusted its implementation based on the fact that hasattr(select, 'poll') was True at that time.

The first solution that comes to mind would be to import eventlet way earlier in the project, before any import to multiprocessing and selectors, see http://eventlet.net/doc/patching.html#monkeypatching-the-standard-library . But I couldn't find the perfect place for this.

A cleaner solution (validated locally) is to import eventlet.queue.Queue instead of multiprocessing.Queue as we use this queue only to synchronise with en eventlet.greenthread. See http://eventlet.net/doc/modules/queue.html . So basically:

--- a/networking_arista/ml2/mechanism_arista.py
+++ b/networking_arista/ml2/mechanism_arista.py
[...]
-from multiprocessing import Queue
+from eventlet.queue import Queue

I'll open a review. Thanks!

PS: I'm hitting this while extending the new OpenStack Neutron API Arista Plugin charm ( https://opendev.org/openstack/charm-neutron-api-plugin-arista ) in order to support the latest OpenStack releases.

[Test Case]
Within the project itself:
https://opendev.org/x/networking-arista/src/branch/master/networking_arista/tests/unit/ml2/test_arista_sync.py#L31 exercises the newly introduced imports.

Within a consuming project:
https://opendev.org/openstack/charm-neutron-api-plugin-arista/src/branch/master/src/tests/tests.yaml#L18 has this code imported by neutron and validates that neutron runs, behaves as expected and succeeds at communicating with an Arista eAPI by creating/deleting networks.

[Regression Potential]
A mistake in the touched code could lead to:
1. neutron-server failing to start if the newly introduced imports don't work.
2. neutron-server failing to communicate with an Arista eAPI if the monkey-patched multiprocessing.Queue doesn't behave as expected. Note that this problem is already there anyway and this is what we intend to fix here.

Changed in networking-arista:
status: New → In Progress
assignee: nobody → Aurelien Lourot (aurelien-lourot)

FYI I pushed debian packages containing the proposed fix to the following PPAs:
https://launchpad.net/~aurelien-lourot/+archive/ubuntu/arista-rocky
https://launchpad.net/~aurelien-lourot/+archive/ubuntu/arista-stein
https://launchpad.net/~aurelien-lourot/+archive/ubuntu/arista-train

This is then validated by a CI system by pluging it an OpenStack setup (Neutron+Nova+Glance+Keystone) connected to an Arista CVX appliance and by creating/deleting networks:
https://review.opendev.org/746871

Reviewed: https://review.opendev.org/743740
Committed: https://git.openstack.org/cgit/x/networking-arista/commit/?id=c859952ee3c0d730e1ffe1cd458dd887bae394b3
Submitter: Zuul
Branch: master

commit c859952ee3c0d730e1ffe1cd458dd887bae394b3
Author: Aurelien Lourot <email address hidden>
Date: Wed Jul 29 13:52:44 2020 +0200

    Fix AttributeError on select.poll()

    Change-Id: Iefcae89cdedd13535c3135ca7227362f1cf9abc8
    Closes-Bug: #1889396

Changed in networking-arista:
status: In Progress → Fix Released
Changed in networking-arista:
assignee: Aurelien Lourot (aurelien-lourot) → nobody
no longer affects: ubuntu
no longer affects: Ubuntu Groovy
Changed in networking-arista (Ubuntu Groovy):
importance: Undecided → High
status: New → Triaged
Changed in networking-arista (Ubuntu Focal):
importance: Undecided → High
status: New → Triaged
Changed in cloud-archive:
importance: Undecided → High
status: New → Triaged
summary: - ML2 plugin fails with "AttributeError: module 'select' has no attribute
- 'poll'"
+ [SRU] ML2 plugin fails with "AttributeError: module 'select' has no
+ attribute 'poll'"
description: updated
description: updated
description: updated

Reviewed: https://review.opendev.org/748763
Committed: https://git.openstack.org/cgit/x/networking-arista/commit/?id=86ec8dd13f792f12157023e1d454277ab95497f3
Submitter: Zuul
Branch: stable/stein

commit 86ec8dd13f792f12157023e1d454277ab95497f3
Author: Aurelien Lourot <email address hidden>
Date: Wed Jul 29 13:52:44 2020 +0200

    Fix AttributeError on select.poll()

    Change-Id: Iefcae89cdedd13535c3135ca7227362f1cf9abc8
    Closes-Bug: #1889396
    Co-Authored-By: Mitchell Jameson <email address hidden>

Reviewed: https://review.opendev.org/748762
Committed: https://git.openstack.org/cgit/x/networking-arista/commit/?id=5a2776f3c0a5da617cd5cd56bb80f2cbcde1ea32
Submitter: Zuul
Branch: stable/train

commit 5a2776f3c0a5da617cd5cd56bb80f2cbcde1ea32
Author: Aurelien Lourot <email address hidden>
Date: Wed Jul 29 13:52:44 2020 +0200

    Fix AttributeError on select.poll()

    Change-Id: Iefcae89cdedd13535c3135ca7227362f1cf9abc8
    Closes-Bug: #1889396
    Co-Authored-By: Mitchell Jameson <email address hidden>

Reviewed: https://review.opendev.org/748761
Committed: https://git.openstack.org/cgit/x/networking-arista/commit/?id=8bc04050a8ca06d2930076d0955e771bed27dae3
Submitter: Zuul
Branch: stable/ussuri

commit 8bc04050a8ca06d2930076d0955e771bed27dae3
Author: Aurelien Lourot <email address hidden>
Date: Wed Jul 29 13:52:44 2020 +0200

    Fix AttributeError on select.poll()

    Change-Id: Iefcae89cdedd13535c3135ca7227362f1cf9abc8
    Closes-Bug: #1889396
    Co-Authored-By: Mitchell Jameson <email address hidden>

tags: added: in-stable-rocky

Reviewed: https://review.opendev.org/748764
Committed: https://git.openstack.org/cgit/x/networking-arista/commit/?id=88322ecc856c6c7d88a527136836e03cc51c9422
Submitter: Zuul
Branch: stable/rocky

commit 88322ecc856c6c7d88a527136836e03cc51c9422
Author: Aurelien Lourot <email address hidden>
Date: Wed Jul 29 13:52:44 2020 +0200

    Fix AttributeError on select.poll()

    Change-Id: Iefcae89cdedd13535c3135ca7227362f1cf9abc8
    Closes-Bug: #1889396
    Co-Authored-By: Mitchell Jameson <email address hidden>

tags: added: in-stable-queens

Reviewed: https://review.opendev.org/748766
Committed: https://git.openstack.org/cgit/x/networking-arista/commit/?id=fc3ca9b01cf01cbd6e0c0c0e0fb83a6c9545a298
Submitter: Zuul
Branch: stable/queens

commit fc3ca9b01cf01cbd6e0c0c0e0fb83a6c9545a298
Author: Aurelien Lourot <email address hidden>
Date: Wed Jul 29 13:52:44 2020 +0200

    Fix AttributeError on select.poll()

    Change-Id: Iefcae89cdedd13535c3135ca7227362f1cf9abc8
    Closes-Bug: #1889396
    Co-Authored-By: Mitchell Jameson <email address hidden>

I suspect a mistake has been done on the Stein backport [0]. I'd like to validate it first locally before proceeding. The other backports look good to me.

0: https://review.opendev.org/#/c/748763

The Stein backport works as intended. Validated with https://github.com/openstack-charmers/zaza-openstack-tests/blob/master/zaza/openstack/charm_tests/neutron_arista/tests.py , which is a test deploying an OpenStack (including Neutron with this plugin), creating/deleting networks and verifying that the networks appear/disappear on the Arista eAPI.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networking-arista - 2019.2.4~git2020070613.c35fef5-0ubuntu2

---------------
networking-arista (2019.2.4~git2020070613.c35fef5-0ubuntu2) groovy; urgency=medium

  * d/p/fix-attribute-error-on-select-poll.patch: Fix AttributeError on
    select.poll() (LP: #1889396).

 -- Corey Bryant <email address hidden> Thu, 03 Sep 2020 10:11:03 -0400

Changed in networking-arista (Ubuntu Groovy):
status: Triaged → Fix Released
Changed in cloud-archive:
status: Triaged → Fix Committed
Corey Bryant (corey.bryant) wrote :

Uploaded to focal unapproved queue, train-staging and stein-staging.

Hello Aurelien, or anyone else affected,

Accepted networking-arista into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/networking-arista/2019.2.3-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in networking-arista (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-focal

I confirm that what's currently in focal-proposed, train-staging and stein-staging fixes the bug.

Validated with https://github.com/openstack-charmers/zaza-openstack-tests/blob/master/zaza/openstack/charm_tests/neutron_arista/tests.py , which is a test deploying an OpenStack (including Neutron with this plugin), creating/deleting networks and verifying that the networks appear/disappear on the Arista eAPI.

tags: added: verification-done-focal
removed: verification-needed-focal
Corey Bryant (corey.bryant) wrote :

Hello Aurelien, or anyone else affected,

Accepted networking-arista into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-ussuri-needed
Corey Bryant (corey.bryant) wrote :

Hello Aurelien, or anyone else affected,

Accepted networking-arista into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-train-needed
Corey Bryant (corey.bryant) wrote :

Hello Aurelien, or anyone else affected,

Accepted networking-arista into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed

Thanks Corey, I also confirm that what's currently in ussuri-proposed, train-proposed and stein-proposed fixes the bug.

Removing the need for my PPAs in our Zaza tests: https://review.opendev.org/752379

tags: added: verification-stein-done verification-train-done verification-ussuri-done
removed: verification-stein-needed verification-train-needed verification-ussuri-needed
tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package networking-arista - 2019.2.3-0ubuntu1

---------------
networking-arista (2019.2.3-0ubuntu1) focal; urgency=medium

  * d/gbp.conf: Create stable/ussuri branch.
  * d/watch: Update path to pypi artifact.
  * d/p/fix-attribute-error-on-select-poll.patch: Fix AttributeError on
    select.poll() (LP: #1889396).
  * New stable point release for OpenStack Ussuri (LP: #1888693).
  * d/p/update-create-network-test-wrapper-to-set-mtu-as-145.patch:
    Fix unit tests failing due to requested MTU too big (LP: #1895059).

 -- Corey Bryant <email address hidden> Wed, 09 Sep 2020 16:50:32 -0400

Changed in networking-arista (Ubuntu Focal):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for networking-arista has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for networking-arista has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package networking-arista - 2019.2.3-0ubuntu0.19.10.1~cloud0
---------------

 networking-arista (2019.2.3-0ubuntu0.19.10.1~cloud0) bionic-train; urgency=medium
 .
   * d/gbp.conf: Create stable/train branch.
   * d/watch: Update path to pypi artifact.
   * d/p/fix-attribute-error-on-select-poll.patch: Fix AttributeError on
     select.poll() (LP: #1889396).
   * New stable point release for OpenStack Train (LP: #1888693).
   * d/control: Align (Build-)Depends with upstream.

Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package networking-arista - 2019.1.9-0ubuntu1~cloud0
---------------

 networking-arista (2019.1.9-0ubuntu1~cloud0) bionic-stein; urgency=medium
 .
   * d/gbp.conf: Create stable/stein branch and align with standard openstack gbp.conf.
   * d/watch: Update path to pypi artifact.
   * d/p/fix-attribute-error-on-select-poll.patch: Fix AttributeError on
     select.poll() (LP: #1889396).
   * New stable point release for OpenStack Stein (LP: #1888693).
   * d/control: Align (Build-)Depends with upstream.
   * d/p/skip-tests.patch: Skip tests that are failing in Ubuntu builds until
     upstream bug is resolved (LP: #1848239)

Corey Bryant (corey.bryant) wrote :

The verification of the Stable Release Update for networking-arista has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package networking-arista - 2019.2.3-0ubuntu1~cloud0
---------------

 networking-arista (2019.2.3-0ubuntu1~cloud0) bionic-ussuri; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 networking-arista (2019.2.3-0ubuntu1) focal; urgency=medium
 .
   * d/gbp.conf: Create stable/ussuri branch.
   * d/watch: Update path to pypi artifact.
   * d/p/fix-attribute-error-on-select-poll.patch: Fix AttributeError on
     select.poll() (LP: #1889396).
   * New stable point release for OpenStack Ussuri (LP: #1888693).
   * d/p/update-create-network-test-wrapper-to-set-mtu-as-145.patch:
     Fix unit tests failing due to requested MTU too big (LP: #1895059).

Changed in cloud-archive:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers