neutron-tempest-iptables_hybrid job failing with internal server error while listining ports

[Slawek Kaplonski]

Example of failed test: http://logs.openstack.org/45/628145/1/gate/neutron-tempest-iptables_hybrid/5b37d22/job-output.txt.gz#_2019-01-03_19_11_27_801081

In neutron-server logs there is error like: http://logs.openstack.org/45/628145/1/gate/neutron-tempest-iptables_hybrid/5b37d22/controller/logs/screen-q-svc.txt.gz?level=ERROR#_Jan_03_19_11_26_174940

There is more examples since last few days: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%2069%2C%20in%20list_ports'%5C%22

[Slawek Kaplonski (slaweq)]

This is very strange issue for me. It looks that it fails in https://github.com/openstack/neutron/blob/master/neutron/services/qos/qos_plugin.py#L102 where there is something like:

    net = network_object.Network.get_object(
        context.get_admin_context(), id=port_res['network_id'])
    if net.qos_policy_id:
        qos_policy = policy_object.QosPolicy.get_network_policy(
            context.get_admin_context(), net.id)

But how it is even possible that there is no network which is configured in port?
From logs it doesn't look like it could be removed in parallel :/

[Slawek Kaplonski (slaweq)]

It's always failing same test:

tempest.api.network.admin.test_ports.PortsAdminExtendedAttrsTestJSON.test_list_ports_binding_ext_attr
or
tempest.api.network.admin.test_ports.PortsAdminExtendedAttrsIpV6TestJSON.test_list_ports_binding_ext_attr

[Slawek Kaplonski (slaweq)]

It looks that this started failing when we switched this job to python3

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/628439

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/628492

[Slawek Kaplonski (slaweq)]

I was able to reproduce this issue locally by running tempest.api.network tests couple of times on vm with Devstack. It happens once for 7 up to 10 runs.

Here is what I found. Problem comes from some network used in different test. GET /ports call in failing test is done by admin user so it will return all ports from all tenants. During preparation of list of ports API worker is trying to get QoS policy for ports. Sometimes it has on list of ports port which was concurrently removed by other API worker (network was from different test). In such case net object in https://github.com/openstack/neutron/blob/master/neutron/services/qos/qos_plugin.py#L102 will be None as this network was already deleted by other thread.

So I think that this is not very common case when it may happen in real life and simple fix by checking if network is not None in https://github.com/openstack/neutron/blob/master/neutron/services/qos/qos_plugin.py#L102 should be enough to fix this issue.

[Slawek Kaplonski (slaweq)]

Patch https://review.openstack.org/628439 was linked to this bug by mistake. It's not related to this one.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/628492
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=65a2f86aafa7637ec07aa0a313ea0fae39758608
Submitter: Zuul
Branch: master

commit 65a2f86aafa7637ec07aa0a313ea0fae39758608
Author: Nate Johnston <email address hidden>
Date: Fri Jan 4 10:28:01 2019 -0500

    Gracefully handle fetch network fail in qos extend port

    When the qos plugin is handling a port resource request through it's
    port resource request extension, sometimes the network a port is
    attached to is looked up and returns None. It may happen like that
    if network will be deleted in concurrent API request.

    Change-Id: Ide4acdf4c373713968f9d43274fb0c7550283c11
    Closes-Bug: #1810504

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b2 development milestone.