stable/xena regression - NotImplementedError: Operator 'getitem' is not supported on this expression

[Corey Bryant]

neutron.tests.unit.objects.test_ports.PortBindingDbObjectTestCase.test_get_duplicated_port_bindings fails on stable/xena with:
NotImplementedError: Operator 'getitem' is not supported on this expression

The code introduced in commit 2fb9ddb01ed44fae19c0184baa8620e95e51e344 [1] depends on a sqlalchemy version that is higher than the minimum version specified in requirements.txt.

[1]

commit 2fb9ddb01ed44fae19c0184baa8620e95e51e344
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Dec 12 05:10:01 2022 +0100

    Delete the PB level registers when deleting the duplicated PB

    The ``neutron-remove-duplicated-port-bindings`` script removes the
    duplicated port binding registers ("ml2_port_bindings" table) that
    have status=INACTIVE.

    This patch also removes the corresponding port binding levels
    ("ml2_port_binding_levels" table) associated to those inactive port
    bindings.

    Closes-Bug: #2000078

    Change-Id: I12fa0764cd0ff509f1859b61060d64cc5a54a7b9
    (cherry picked from commit 572185e1fd252772d8baaec099bf8bedfc837013)
    (cherry picked from commit 07855a68266484ae187c76708a727b8a4db1e2e2)

To recreate:

root@f1:~/neutron# git remote -v
origin https://opendev.org/openstack/neutron (fetch)
origin https://opendev.org/openstack/neutron (push)

root@f1:~/neutron# git branch
* (HEAD detached at 19.7.0)
  master
  stable/xena

root@f1:~/neutron# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal

root@f1:~/neutron# tox -e py3 --notest
  ...
  py3: OK (22.37 seconds)
  congratulations :) (22.46 seconds)

root@f1:~/neutron# .tox/py3/bin/pip3 freeze | grep -i -E 'oslo.db|sqlalchemy'
oslo.db==11.0.0
SQLAlchemy==1.4.23
sqlalchemy-migrate==0.13.0

root@f1:~/neutron# grep -i -E 'oslo.db|sqlalchemy' requirements.txt
SQLAlchemy>=1.3.23 # MIT
oslo.db>=4.44.0 # Apache-2.0

root@f1:~/neutron# .tox/py3/bin/pip3 install oslo.db==4.44.0 SQLAlchemy==1.3.23
...
Successfully installed SQLAlchemy-1.3.23 oslo.db-4.44.0

root@f1:~/neutron# tox -e py3 -- neutron.tests.unit.objects.test_ports.PortBindingDbObjectTestCase.test_get_duplicated_port_bindings
...
{0} neutron.tests.unit.objects.test_ports.PortBindingDbObjectTestCase.test_get_duplicated_port_bindings [0.367953s] ... FAILED [117/27872]

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/root/neutron/neutron/tests/base.py", line 183, in func
    return f(self, *args, **kwargs)

      File "/root/neutron/neutron/tests/unit/objects/test_ports.py", line 68, in test_get_duplicated_port_bindings
    dup_pb = ports.PortBinding.get_duplicated_port_bindings(self.context)

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1012, in wrapper
    return fn(*args, **kwargs)

      File "/root/neutron/neutron/objects/ports.py", line 115, in get_duplicated_port_bindings
    sqlalchemy.select(cls.db_model.port_id).

      File "<string>", line 2, in select

      File "<string>", line 2, in __init__

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py", line 139, in warned
    return fn(*args, **kwargs)

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/sql/selectable.py", line 3114, in __init__
    for c in columns:

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 432, in __getitem__
    return self.operate(getitem, index)

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/orm/attributes.py", line 226, in operate
    return op(self.comparator, *other, **kwargs)

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 432, in __getitem__
    return self.operate(getitem, index)

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/orm/properties.py", line 367, in operate
    return op(self.__clause_element__(), *other, **kwargs)

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 432, in __getitem__
    return self.operate(getitem, index)

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 762, in operate
    return op(self.comparator, *other, **kwargs)

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/sql/operators.py", line 432, in __getitem__
    return self.operate(getitem, index)

      File "<string>", line 1, in <lambda>

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/sql/type_api.py", line 67, in operate
    return o[0](self.expr, op, *(other + o[1:]), **kwargs)

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/sql/default_comparator.py", line 237, in _getitem_impl
    _unsupported_impl(expr, op, other, **kw)

      File "/root/neutron/.tox/py3/lib/python3.8/site-packages/sqlalchemy/sql/default_comparator.py", line 241, in _unsupported_impl
    raise NotImplementedError(

    NotImplementedError: Operator 'getitem' is not supported on this expression

[Corey Bryant (corey.bryant)]

Hello Corey, or anyone else affected,

Accepted neutron into xena-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:xena-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-xena-needed to verification-xena-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-xena-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Corey:

"openstack/requirements", more in particular "upper-constraints.txt" file, define the maximum versions to be used, not the minimum. I think that was a typo in the description.

The max version in Xena is 1.4.23. The max version in Wallaby is 1.3.23. I think you are referring to Wallaby, not to Xena.

The "select()" function accepts column arguments positionally since 1.4.x [3]. I'll push a patch for Wallaby.

Regards.

[1]https://github.com/openstack/requirements/blob/stable/xena/upper-constraints.txt#L157
[2]https://github.com/openstack/requirements/blob/stable/wallaby/upper-constraints.txt#L152
[3]https://docs.sqlalchemy.org/en/20/core/selectable.html#sqlalchemy.sql.expression.select

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Cory:

Actually the Wallaby patch had this problem in mind: https://review.opendev.org/c/openstack/neutron/+/877493/2/neutron/objects/ports.py

Please check your env and the requirements file downloaded.

Regards.

[Corey Bryant (corey.bryant)]

Thanks for the pointer. I'll take a look at the wallaby patch. Re: upper-constraints. I was referring to the minimum version specified in requirements.txt rather than upper-constraints.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

You are right [1], I'll push a patch for Xena.

[1]https://github.com/openstack/neutron/blob/43ee9f3a9474f382fba9db063722d1c0dfee9b48/requirements.txt#L22

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/883288

[Corey Bryant (corey.bryant)]

Hello Corey, or anyone else affected,

Accepted neutron into xena-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:xena-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-xena-needed to verification-xena-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-xena-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/883288
Committed: https://opendev.org/openstack/neutron/commit/4372dd097a23ff7f042b3dfc3bf14880861b6a8f
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 4372dd097a23ff7f042b3dfc3bf14880861b6a8f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue May 16 18:48:25 2023 +0200

    [stable-only] Don't use "sqlalchemy.select" with SQLAlchemy<1.4

    The "sqlalchemy.select" function accepts column arguments positionally
    since 1.4.x [3]. However, the minimum version accepted in Xena branch
    is 1.3.23 [2]. This patch uses the implementation merged in Wallaby
    branch [3] for the method "get_duplicated_port_bindings", that is
    compatible with SQLAlchemy 1.3 and 1.4.

    [1]https://docs.sqlalchemy.org/en/20/core/selectable.html#sqlalchemy.sql.expression.select
    [2]https://github.com/openstack/requirements/blob/stable/xena/upper-constraints.txt#L157
    [3]https://review.opendev.org/c/openstack/neutron/+/877493

    Closes-Bug: #2019859
    Change-Id: I9e1e43975fb616b72f83489f34c573674a03a698

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:19.7.0-0ubuntu1~cloud1
---------------

 neutron (2:19.7.0-0ubuntu1~cloud1) focal-xena; urgency=medium
 .
   * d/p/bug-2000078-revert-xena-fix.patch,
     d/p/bug-2000078-apply-wallaby-fix.patch: Revert the upstream xena
     patch that depends on a more recent version of sqlalchemy than is
     available in the xena cloud archive. Apply the corresponding upstream
     wallaby that is compatible with sqlalchemy in the xena cloud archive
     (LP: #2019859).
 .
 neutron (2:19.7.0-0ubuntu1~cloud0) focal-xena; urgency=medium
 .
   * New stable point release for OpenStack Xena (LP: #2019762).
   * d/p/load-required-test-config.patch: Fix failing unit tests.
   * d/p/revert-bug-2000078-fix.patch: Revert patch that depends on
     more recent version of sqlalchemy (LP: #2019859).

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Patch https://review.opendev.org/c/openstack/neutron/+/883288 merged, moving the status to Fix Committed.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron xena-eom release.