neutron_tempest_plugin.api sorted() Function Issue

[Ayhan ilhan]

Hi,

We got mismatch error in some tests as below. I debugged it and the test passes if we change sorted() function. Because sorted() function sorts by ascii character. When sorted in ASCII order, digits are followed by uppercase characters and then lowercase characters. If I have a network name starting with a capital letter, it gives an error.

If we change the sorted() function as follows, the test passed.
expected = sorted(retrieved_names, key=lambda x: (x.isdigit(), x.upper(), x.lower()))

Example tests:
neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_sorts_asc
neutron_tempest_plugin.api.test_networks.NetworksSearchCriteriaTest.test_list_sorts_desc
neutron_tempest_plugin.api.test_subnets.SubnetsSearchCriteriaTest.test_list_sorts_asc
neutron_tempest_plugin.api.test_subnets.SubnetsSearchCriteriaTest.test_list_sorts_desc
neutron_tempest_plugin.api.test_security_groups.SecGroupSearchCriteriaTest.test_list_sorts_by_name_asc
neutron_tempest_plugin.api.test_security_groups.SecGroupSearchCriteriaTest.test_list_sorts_by_name_desc

File: /etc/tempest/neutron-tempest-plugin/neutron_tempest_plugin/api/base.py
```
...
    def _test_list_sorts(self, direction):
        sort_args = {
            'sort_dir': direction,
            'sort_key': self.field
        }
        body = self.list_method(**sort_args)
        resources = self._extract_resources(body)
        self.assertNotEmpty(
            resources, "%s list returned is empty" % self.resource)
        retrieved_names = [res[self.field] for res in resources]
        expected = sorted(retrieved_names)
        if direction == constants.SORT_DIRECTION_DESC:
            expected = list(reversed(expected))
        self.assertEqual(expected, retrieved_names)
...
```
Debug:

retrieved_names
['', '', '', '123test', 'abc1', 'admin-xxxxx-xxxxx-net-01', 'admin-xxxxx-xxxxx-net-01', 'xxxxx-xxxx-net', 'HA network tenant 2b09e7b2e1244fd4be2dc4092e39d17b', 'lb-xxxx-net', 'mxxxx-temp-xxxx-net', 'mxxx-xxx-xxx-net', 'mxxx-xxxx-xxxx-net', 'port-search-test-net', 'subnet-search-test-net', 'subnet-search-test-net', 'subnet-search-test-net', 'tempest-bgp-net', 'tempest-internal-dns-test-network-1876397349', 'tempest-internal-dns-test-network-332103665', 'tempest-MetadataTest-2030842204', 'tempest-test-network--1146463009', 'tempest-test-network--1164657513', 'tempest-test-network--1229813769', 'tempest-test-network--1233298883', 'tempest-test-network--1331054017', 'tempest-test-network--1415812209', 'tempest-test-network--1420967741', 'tempest-test-network--1424837927', 'tempest-test-network--1522800460', 'tempest-test-network--1542595587', 'tempest-test-network--1602874451', 'tempest-test-network--1651416667', 'tempest-test-network--1756609168', 'tempest-test-network--2053127518', 'tempest-test-network--2065848286', 'tempest-test-network--2086276043', 'tempest-test-network--234088028', 'tempest-test-network--303087501', 'tempest-test-network--377750320', 'tempest-test-network--433648246', 'tempest-test-network--488800423', 'tempest-test-network--524111130', 'tempest-test-network--586734631', 'tempest-test-network--602944876', 'tempest-test-network--653992987', 'tempest-test-network-884004988', 'test1', 'test1', 'test10', 'testproject-other-net']
(Pdb) print(sorted(retrieved_names))

print(sorted(retrieved_names))
['', '', '', '123test', 'HA network tenant 2b09e7b2e1244fd4be2dc4092e39d17b', 'abc1', 'admin-xxxxx-xxxxx-net-01', 'admin-vprotect-provider-net-01', 'devxxx-xxx-net', 'lxxx-xxx-net', 'mxxx-xxxx-xxxxx-net', 'mxxx-xxxx-xxxxx-net', 'mxxx-xxx-xxxx-net', 'port-search-test-net', 'subnet-search-test-net', 'subnet-search-test-net', 'subnet-search-test-net', 'tempest-MetadataTest-2030842204', 'tempest-bgp-net', 'tempest-internal-dns-test-network-1876397349', 'tempest-internal-dns-test-network-332103665', 'tempest-test-network--1146463009', 'tempest-test-network--1164657513', 'tempest-test-network--1229813769', 'tempest-test-network--1233298883', 'tempest-test-network--1331054017', 'tempest-test-network--1415812209', 'tempest-test-network--1420967741', 'tempest-test-network--1424837927', 'tempest-test-network--1522800460', 'tempest-test-network--1542595587', 'tempest-test-network--1602874451', 'tempest-test-network--1651416667', 'tempest-test-network--1756609168', 'tempest-test-network--2053127518', 'tempest-test-network--2065848286', 'tempest-test-network--2086276043', 'tempest-test-network--234088028', 'tempest-test-network--303087501', 'tempest-test-network--377750320', 'tempest-test-network--433648246', 'tempest-test-network--488800423', 'tempest-test-network--524111130', 'tempest-test-network--586734631', 'tempest-test-network--602944876', 'tempest-test-network--653992987', 'tempest-test-network-884004988', 'test1', 'test1', 'test10', 'testproject-other-net']

Regards,

[Martin Kopec (mkopec)]

this is an issue on neutron-tempest-plugin's side, not tempest, i'll change the component to neutron.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/905853

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/905853
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/71a73245083f0135aea2fb063fd3dd7332ff08e1
Submitter: "Zuul (22348)"
Branch: master

commit 71a73245083f0135aea2fb063fd3dd7332ff08e1
Author: Martin Kopec <email address hidden>
Date: Wed Jan 17 12:02:24 2024 +0100

    Fix network sorting in API tests

    There is a mismatch in some tests because sorted() sorts in ASCII
    order - digits are followed by uppercase characters and then
    lowercase characters. That doesn't work if a network's name starts
    with a capital letter.

    Closes-Bug: #2049211
    Change-Id: I8328d7d8ca7b49b99feb24d77525abdc064aae80

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin 2.6.0 release.