250 unit test failures under Python 3.11

Bug #2000436 reported by Thomas Goirand
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Committed
High
Eric Harney

Bug Description

Trying to build Cinder 21.0.0 (Zed) under Python 3.11 in Debian unstable results in 250 unit test failures the way I cut/past below.

This looks like a miss-use of the mock object that now fails. See the doc from upstream Python:

bpo-43478: Mocks can no longer be provided as the specs for other Mocks. As a result, an already-mocked object cannot be passed to mock.Mock(). This can uncover bugs in tests since these Mock-derived Mocks will always pass certain tests (e.g. isinstance) and builtin assert functions (e.g. assert_called_once_with) will unconditionally pass.

python/cpython#87644
https://docs.python.org/3.11/whatsnew/3.11.html

Please help me fix this so that Cinder can stay in Debian and be part of Debian 12 (Bookworm).

FAIL: cinder.tests.unit.volume.drivers.ceph.test_rbd_iscsi.RBDISCSITestCase.test_unsupported_version
cinder.tests.unit.volume.drivers.ceph.test_rbd_iscsi.RBDISCSITestCase.test_unsupported_version
----------------------------------------------------------------------
testtools.testresult.real._StringException: pythonlogging:'': {{{
2022-12-25 17:15:37,860 INFO [keystonemiddleware.auth_token] Starting Keystone auth_token middleware
2022-12-25 17:15:37,867 WARNING [keystonemiddleware._common.config] The option "auth_url" is not known to keystonemiddleware
2022-12-25 17:15:37,867 WARNING [keystonemiddleware.auth_token] AuthToken middleware is set with keystone_authtoken.service_token_roles_required set to False. This is backwards compatible but deprecated behaviour. Please set this to True.
2022-12-25 17:15:37,872 WARNING [keystonemiddleware.auth_token] Configuring www_authenticate_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin endpoint
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.11/unittest/mock.py", line 1369, in patched
    return func(*newargs, **newkeywargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/<<PKGBUILDDIR>>/cinder/tests/unit/volume/drivers/ceph/test_rbd_iscsi.py", line 167, in test_unsupported_version
    self.setup_mock_client()
  File "/usr/lib/python3.11/unittest/mock.py", line 1366, in patched
    with self.decoration_helper(patched,
  File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/unittest/mock.py", line 1348, in decoration_helper
    arg = exit_stack.enter_context(patching)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/contextlib.py", line 502, in enter_context
    result = _enter(cm)
             ^^^^^^^^^^
  File "/usr/lib/python3.11/unittest/mock.py", line 1491, in __enter__
    new = Klass(**_kwargs)
          ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/unittest/mock.py", line 2100, in __init__
    _safe_super(MagicMixin, self).__init__(*args, **kw)
  File "/usr/lib/python3.11/unittest/mock.py", line 1100, in __init__
    _safe_super(CallableMixin, self).__init__(
  File "/usr/lib/python3.11/unittest/mock.py", line 451, in __init__
    self._mock_add_spec(spec, spec_set, _spec_as_instance, _eat_self)
  File "/usr/lib/python3.11/unittest/mock.py", line 502, in _mock_add_spec
    raise InvalidSpecError(f'Cannot spec a Mock object. [object={spec!r}]')
unittest.mock.InvalidSpecError: Cannot spec a Mock object. [object=<MagicMock name='mock.client.RBDISCSIClient' id='140512319587344'>]

Tags: tests
Revision history for this message
Sofia Enriquez (lsofia-enriquez) wrote :

Hi Thomas Goirand, hope this message finds you well.
I'll bring this up on the cinder bug meeting. Would you mind sharing the full list of 250 unit test that are failing so i can reproduce in my env?
Thanks in advance,
Sofia

tags: added: tests
Changed in cinder:
importance: Undecided → High
Revision history for this message
Thomas Goirand (thomas-goirand) wrote :
Download full text (27.6 KiB)

Sure ! Here's the list:

FAIL: cinder.tests.unit.backup.drivers.test_backup_ceph.BackupCephTestCase.test_backup_volume_from_rbd
FAIL: cinder.tests.unit.backup.drivers.test_backup_ceph.BackupCephTestCase.test_backup_volume_from_rbd_fail
FAIL: cinder.tests.unit.backup.drivers.test_backup_ceph.BackupCephTestCase.test_backup_volume_from_rbd_fail2
FAIL: cinder.tests.unit.backup.drivers.test_backup_ceph.BackupCephTestCase.test_piped_execute
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test__get_existing_volume_ref_name
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test__safe_hostname_1_True
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test__safe_hostname_2_False
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test__safe_hostname_unique_1_True
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test__safe_hostname_unique_2_False
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test__safe_hostname_unique_without_initiator_1_True
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test__safe_hostname_unique_without_initiator_2_False
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_backup_iscsi_volume_with_chap_disabled
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_backup_iscsi_volume_with_chap_enabled
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_clone_volume_with_vvs
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_clone_iscsi_volume_with_chap_disabled
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_cloned_qos_volume
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_cloned_volume_1_volume
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_cloned_volume_2_volume_name_id
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_cloned_volume_offline_copy
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_group
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_group_from_src
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_group_from_src_group
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_group_snapshot
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_group_with_replication_enabled
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_group_with_replication_from_src_group
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_snapshot_1___snapshot____osv_dh_F5VGRTseuujPjbeRBVg__
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3par.TestHPE3PARDriverBase.test_create_snapshot_2___snapshot_name_id____osv_L4I73ONuTci9Fd4ceij_MQ__
FAIL: cinder.tests.unit.volume.drivers.hpe.test_hpe3pa...

Changed in cinder:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to cinder (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/869258

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/869396

Eric Harney (eharney)
Changed in cinder:
status: Triaged → In Progress
assignee: nobody → Eric Harney (eharney)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to cinder (master)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/867824
Committed: https://opendev.org/openstack/cinder/commit/8d9e0bf976d9d367e203a6ba9a0ddd82b73d6b20
Submitter: "Zuul (22348)"
Branch: master

commit 8d9e0bf976d9d367e203a6ba9a0ddd82b73d6b20
Author: Eric Harney <email address hidden>
Date: Thu Dec 15 10:47:14 2022 -0500

    Tests: Fix double mock of Popen (test_backup_ceph)

    This is an error in Python 3.11's mock. Stop mocking
    Popen when it is already mocked by @common_mocks.

    Related-Bug: #2000436
    Change-Id: I5c1a4cd0ba0394849987203888b942169550672e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/c/openstack/cinder/+/869396
Committed: https://opendev.org/openstack/cinder/commit/c75b3260c844464829bcbc18460684cc69e89259
Submitter: "Zuul (22348)"
Branch: master

commit c75b3260c844464829bcbc18460684cc69e89259
Author: Eric Harney <email address hidden>
Date: Thu Jan 5 14:29:36 2023 -0500

    test_rbd_iscsi: Make tests compatible with python 3.11

    This makes these tests work in Python 3.11.

    Also includes a few cleanups such as reducing storing
    the driver in "self", and removal of unneeded fakes code
    for rbd_iscsi_client.

    Related-Bug: #2000436
    Change-Id: I471ce2887a9884a5478a303e4b4db9cf8e55b6fb

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/c/openstack/cinder/+/869258
Committed: https://opendev.org/openstack/cinder/commit/52981bbb2767b90e2b0b150829df4c30a60557d9
Submitter: "Zuul (22348)"
Branch: master

commit 52981bbb2767b90e2b0b150829df4c30a60557d9
Author: Eric Harney <email address hidden>
Date: Wed Jan 4 14:34:38 2023 -0500

    Run py311 unit tests (non-voting)

    Add a non-voting gate job for py311 unit tests.

    Related-Bug: #2000436
    Depends-On: I5c1a4cd0ba0394849987203888b942169550672e
    Depends-On: I471ce2887a9884a5478a303e4b4db9cf8e55b6fb
    Depends-On: I4fdf57755b25bbdd19013fd3a2fd86a7d1dd29e9
    Change-Id: I06741d95050b42e1801260b1b8d1806c9d3bf6c9

Eric Harney (eharney)
Changed in cinder:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.