test_port_list_filter_by_ip has inaccurate assertions

[Phil Sphicas]

Background:

The test tempest.api.network.test_ports.PortsTestJSON.test_port_list_filter_by_ip is intended to validate the Neutron "List ports" API call, with the a filter applied to the fixed_ips field.

Per the Neutron API docs for list ports:

Default policy settings return only those ports that are owned by the project of the user who submits the request, unless the request is submitted by a user with administrative rights.[0]

The Neutron API supports filtering based on all top level attributes of a resource. Filters are applicable to all list requests.[1]

Therefore, the list of ports returned by the API should be any ports visible to the current user, possibly in multiple projects, where the IP address of every returned ports matches the filter.

The current implementation of the test checks for the following:

#1. That ports from only one tenant are in the response[2]
#2. That the created port is in the list of ports[3]
#3. That the IP of the create port is in the list of IPs[4]
#4. That the network of the created port is in the list of networks associated with the returned ports[5]

Issue:

#1 can result in an error if running with admin credentials (such as the ones created by tempest account-generator --with-admin)
#2 is valid
#3 is redundant if #2 passes
#4 is redundant if #2 passes

More importantly, these checks do not properly validate that the filtering is working. Even if the full list of ports were returned, the test would pass.

Recommendation:

The test should just ensure that the port that we created is in the list, and that all returned ports match the filter.

References:
[0]: https://docs.openstack.org/api-ref/network/v2/?expanded=list-ports-detail#list-ports
[1]: https://wiki.openstack.org/wiki/Neutron/APIv2-specification#Filtering_and_Column_Selection
[2]: https://github.com/openstack/tempest/blob/d3155db773dc83f87a184eb76a4976df2270dfbb/tempest/api/network/test_ports.py#L188-L189
[3]: https://github.com/openstack/tempest/blob/d3155db773dc83f87a184eb76a4976df2270dfbb/tempest/api/network/test_ports.py#L197
[4]: https://github.com/openstack/tempest/blob/d3155db773dc83f87a184eb76a4976df2270dfbb/tempest/api/network/test_ports.py#L198
[5]: https://github.com/openstack/tempest/blob/d3155db773dc83f87a184eb76a4976df2270dfbb/tempest/api/network/test_ports.py#L199

Example:

This is an example of a failure associated with #1:

    Traceback (most recent call last):
      File "/home/ubuntu/code_base/tempest/tempest/api/network/test_ports.py", line 189, in test_port_list_filter_by_ip
        'Ports from multiple tenants are in the list resp')
      File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 411, in assertEqual
        self.assertThat(observed, matcher, message)
      File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 498, in assertThat
        raise mismatch_error
    testtools.matchers._impl.MismatchError: 2 != 1: Ports from multiple tenants are in the list resp

From this response:

    {
      "ports": [
        {
          "status": "ACTIVE",
          "binding:host_id": "mtn57r03c002",
          "description": "",
          "allowed_address_pairs": [],
          "tags": [],
          "extra_dhcp_opts": [],
          "updated_at": "2019-08-15T22:59:15Z",
          "device_owner": "compute:mtn57a-kvm-az01",
          "revision_number": 9,
          "port_security_enabled": true,
          "binding:profile": {},
          "fixed_ips": [
            {
              "subnet_id": "843498e2-4d8d-434f-8d8c-5fd88de59710",
              "ip_address": "172.25.0.6"
            }
          ],
          "id": "64feda4e-48d7-4068-be2c-341fe5d002c8",
          "security_groups": [
            "5c5656fa-ab67-4759-b6ea-c2003142b682"
          ],
          "device_id": "ce8f107c-cb26-4251-a161-cad5bbdabd43",
          "name": "",
          "admin_state_up": true,
          "network_id": "a0ddf649-a10e-44cb-a678-cd54b50e5736",
          "tenant_id": "af1576634c8047baa6e81f41565549a1",
          "binding:vif_details": {
            "port_filter": true,
            "ovs_hybrid_plug": true
          },
          "binding:vnic_type": "normal",
          "binding:vif_type": "ovs",
          "mac_address": "fa:16:3e:51:6b:b7",
          "project_id": "af1576634c8047baa6e81f41565549a1",
          "created_at": "2019-08-15T22:58:26Z"
        },
        {
          "status": "DOWN",
          "binding:host_id": "",
          "description": "",
          "allowed_address_pairs": [],
          "tags": [],
          "extra_dhcp_opts": [],
          "updated_at": "2019-08-15T23:00:22Z",
          "device_owner": "",
          "revision_number": 5,
          "port_security_enabled": true,
          "binding:profile": {},
          "fixed_ips": [
            {
              "subnet_id": "92280012-b3fc-46c7-95ab-ad31794755a0",
              "ip_address": "172.25.0.6"
            }
          ],
          "id": "b0677ab7-57c6-4a5f-8b94-654dcd8dd663",
          "security_groups": [
            "6546ca76-704a-4895-95df-8e85f7a5b514"
          ],
          "device_id": "",
          "name": "",
          "admin_state_up": true,
          "network_id": "b7cbdbbd-42ef-4f96-802c-1077d4a6a3c4",
          "tenant_id": "a0abca66bf2f4c58b47d8913a639f3d5",
          "binding:vif_details": {},
          "binding:vnic_type": "normal",
          "binding:vif_type": "unbound",
          "mac_address": "fa:16:3e:19:32:57",
          "project_id": "a0abca66bf2f4c58b47d8913a639f3d5",
          "created_at": "2019-08-15T23:00:22Z"
        }
      ]
    }

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/676852

[Ghanshyam Mann (ghanshyammann)]

Test run with non-admin users which should not be allowed to get the other tenants ports. are you running with other cred and overriden RBAC rule ?

[Phil Sphicas (ps3910)]

This is was with pre-provisioned credentials in accounts.yaml that do have the admin role. The tenants are initially empty, but the conflicting resources are created by parallel workers.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/676852
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=0ff7e3d7204cb5df539f470e759738957ef0c70b
Submitter: Zuul
Branch: master

commit 0ff7e3d7204cb5df539f470e759738957ef0c70b
Author: Sphicas, Phil (ps3910) <email address hidden>
Date: Thu Aug 15 20:34:02 2019 -0700

    Update test_port_list_filter_by_ip pass conditions

    This patchset updates the pass assertions to check that the created
    port is in the list of ports returned, and that every port returned
    has a matching fixed IP.

    The check that all the ports in the response are from a single tenant
    is retained, so this test is expected to sometimes fail if run with
    admin credentials.

    Change-Id: Ia20af71d031504aa08fc0ea92fd35cbc408edba3
    Partial-Bug: #1840401

[Martin Kopec (mkopec)]

Marking as Fixed Released as the associated patch got merged and it changed assertions within the test, so I assume that the assertions are accurate now -> the bug is fixed.