many unit tests issues with objects compared to strings

[Thomas Goirand]

Running unit tests of Horizon in Debian Unstable leads to many unit test failures like this one below.

My instinct tells me that this is Python 3.12 related, but I'm not sure. Here's a pip freeze output with the installed build-dependency of the package:

alabaster==0.7.12
appdirs==1.4.4
asgiref==3.7.2
async-timeout==4.0.3
attrs==23.1.0
autopage==0.4.0
Babel==2.10.3
blinker==1.6.2
calmjs==3.4.2
calmjs.parse==1.2.5
calmjs.types==1.0.1
certifi==2023.7.22
chardet==5.2.0
charset-normalizer==3.2.0
cliff==4.2.0
cmd2==2.4.3+ds
coverage==7.2.7
cryptography==38.0.4
csscompressor==0.9.5
debtcollector==2.5.0
decorator==5.1.1
Deprecated==1.2.14
deprecation==2.0.7
Django==4.2.5
django-appconf==1.0.5
django-compressor==4.0
django-debreach==2.1.0
django-pyscss==2.0.2
dnspython==2.4.2
docutils==0.19
dogpile.cache==1.1.8
dulwich==0.21.6
enmerkar==0.7.1
eventlet==0.33.1
exceptiongroup==1.1.3
execnet==2.0.0
extras==1.0.0
fasteners==0.17.3
fixtures==4.0.1
flake8==5.0.4
freezegun==1.2.1
futurist==2.4.1
greenlet==2.0.2
h11==0.14.0
hacking==4.1.0
idna==3.3
imagesize==1.4.1
importlib-metadata==4.12.0
iniconfig==1.1.1
iso8601==1.0.2
jaraco.classes==3.2.1
jeepney==0.8.0
Jinja2==3.1.2
jmespath==1.0.1
jsonpatch==1.32
jsonpointer==2.3
jsonschema==4.10.3
keyring==24.2.0
keystoneauth1==5.3.0
lxml==4.9.3
Mako==1.2.4.dev0
MarkupSafe==2.1.3
mccabe==0.7.0
monotonic==1.6
more-itertools==10.1.0
msgpack==1.0.3
netaddr==0.8.0
netifaces==0.11.0
oauthlib==3.2.2
openstackdocstheme==1.20.0
openstacksdk==1.0.1
os-client-config==2.1.0
os-service-types==1.7.0
osc-lib==2.8.1
oslo.concurrency==5.1.1
oslo.config==9.1.1
oslo.context==5.1.1
oslo.i18n==6.0.0
oslo.log==5.2.0
oslo.policy==4.1.1
oslo.serialization==5.1.1
oslo.upgradecheck==2.1.1
oslo.utils==6.1.0
osprofiler==3.4.3
outcome==1.2.0
packaging==23.1
pbr==5.11.1
pep8==1.7.1
pluggy==1.3.0
ply==3.11
prettytable==3.6.0
pycodestyle==2.10.0
pyflakes==2.5.0
Pygments==2.15.1
pyinotify==0.9.6
PyJWT==2.7.0
pymongo==3.11.0
pyOpenSSL==23.0.0
pyparsing==3.1.1
pyperclip==1.8.2
pyrsistent==0.18.1
pyScss==1.4.0
pytest==7.4.2
pytest-django==4.5.2
pytest-xdist==3.3.1
python-cinderclient==9.3.0
python-dateutil==2.8.2
python-glanceclient==4.3.0
python-keystoneclient==5.1.0
python-memcached==1.58
python-neutronclient==9.0.0
python-novaclient==18.3.0
python-swiftclient==4.2.0
pytz==2023.3.post1
PyYAML==6.0.1
rcssmin==1.1.0
redis==4.3.4
requests==2.31.0
requestsexceptions==1.4.0
rfc3986==1.5.0
rjsmin==1.2.0
roman==3.3
SecretStorage==3.3.3
selenium==4.13.0
semantic-version==2.9.0
simplejson==3.19.1
six==1.16.0
sniffio==1.2.0
snowballstemmer==2.2.0
sortedcontainers==2.4.0
Sphinx==5.3.0
sqlparse==0.4.2
stevedore==5.1.0
testscenarios==0.5.0
testtools==2.5.0
trio==0.22.2
trio-websocket==0.10.3
urllib3==1.26.16
warlock==2.0.1
wcwidth==0.2.5
WebOb==1.8.6
wrapt==1.14.1
wsproto==1.2.0
XStatic==1.0.3
XStatic-Angular==1.8.2.2
XStatic-Angular-Bootstrap==2.5.0.0
XStatic-Angular-FileUpload==12.0.4.0
XStatic-Angular-Gettext==2.4.1.0
XStatic-Angular-lrdragndrop==1.0.2.2
XStatic-Angular-Schema-Form==0.8.13.0
XStatic-angular-ui-router==0.3.1.4
XStatic-Bootstrap-Datepicker==1.3.1.0
XStatic-Bootstrap-SCSS==3.4.1.0
XStatic-bootswatch==3.3.7.0
XStatic-D3==3.5.17.0
XStatic-Font-Awesome==4.7.0.0
XStatic-Hogan==2.0.0.2
XStatic-Jasmine==2.4.1.0
XStatic-jQuery==3.5.1.0
XStatic-JQuery-Migrate==3.3.2.1
XStatic-jquery-ui==1.12.0.1
XStatic-JQuery.quicksearch==2.0.4.1
XStatic-JQuery.TableSorter==2.14.5.1
XStatic-JSEncrypt==2.3.1.1
XStatic-Magic-Search==0.2.5.1
XStatic-mdi==1.6.50.2
XStatic-objectpath==1.2.1.0
XStatic-Rickshaw==1.5.0.2
XStatic-roboto-fontface==0.5.0.0
XStatic-smart-table==1.4.13.2
XStatic-Spin==1.2.8.2
XStatic-term.js==0.0.7.0
XStatic-tv4==1.2.7.0
xvfbwrapper==0.2.9
zipp==1.0.0

and here's a typical failure below. Note that there's maybe more than 3 dozen of issues like it:

__________________ WorkflowsTests.test_workflow_registration ___________________
[gw3] linux -- Python 3.11.5 /usr/bin/python3.11

self = <horizon.test.unit.workflows.test_workflows.WorkflowsTests testMethod=test_workflow_registration>

    def test_workflow_registration(self):
        req = self.factory.get("/foo")
        flow = WorkflowForTesting(req)
> self.assertQuerysetEqual(flow.steps,
                                 ['<StepOne: action_one>',
                                  '<StepTwo: action_two>'])

horizon/test/unit/workflows/test_workflows.py:328:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python3/dist-packages/django/test/testcases.py:1330: in assertQuerysetEqual
    return self.assertQuerySetEqual(*args, **kw)
/usr/lib/python3/dist-packages/django/test/testcases.py:1346: in assertQuerySetEqual
    return self.assertEqual(list(items), values, msg=msg)
E AssertionError: Lists differ: [<StepOne: action_one>, <StepTwo: action_two>] != ['<StepOne: action_one>', '<StepTwo: action_two>']
E
E First differing element 0:
E <StepOne: action_one>
E '<StepOne: action_one>'
E
E - [<StepOne: action_one>, <StepTwo: action_two>]
E + ['<StepOne: action_one>', '<StepTwo: action_two>']
E ? + + + +

Probably, it used to be that Python returned strings, but now it's not? Anyways, calling repr() on each object fixes it, with something like this:

       self.assertQuerysetEqual(map(repr, flow.steps),
                                 ['<StepOne: action_one>',
                                  '<StepTwo: action_two>'])

though, this breaks in earlier environment, as a map isn't a list, so this really fixes all cases:

       self.assertQuerysetEqual(list(map(repr, flow.steps)),
                                 ['<StepOne: action_one>',
                                  '<StepTwo: action_two>'])

My patch here fixes them all:
https://review.opendev.org/c/openstack/horizon/+/897310

Cheers,

Thomas

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/horizon/+/897310
Committed: https://opendev.org/openstack/horizon/commit/8770753fcc8f4c224c35b761d64174c0081cad00
Submitter: "Zuul (22348)"
Branch: master

commit 8770753fcc8f4c224c35b761d64174c0081cad00
Author: Thomas Goirand <email address hidden>
Date: Wed Oct 4 11:44:00 2023 +0200

    Specify transform=repr in assertQuerysetEqual()

    Previously "repr" was automatically applied to "qs" argument of
    assertQuerysetEqual() and most horizon unittest behaviors assume it.
    It was deprecated in Django 3.2 and removed in Django 4.1. We need to
    specify transform=repr explicitly to use the existing behavior.

    [1] https://docs.djangoproject.com/en/3.2/topics/testing/tools/#django.test.TransactionTestCase.assertQuerysetEqual

    Closes-Bug: #2038474
    Change-Id: Ie7c7e9a1efc492889639e25509c8e614268c1d26

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/horizon 23.4.0 release.