neutron

QoS plugin fails if network is not found

Bug #1811455 reported by Michael Johnson
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
In Progress
Undecided
Michael Johnson

Bug Description

Master neutron (Stein):
We are intermittently seeing gate failures with a q-svc exception:

Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server [None req-9e4027ef-c0b5-4d46-99be-1a1da640c506 None None] Exception during message handling: AttributeError: 'NoneType' object has no attribute 'qos_policy_id'
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/api/rpc/handlers/dhcp_rpc.py", line 146, in get_active_networks_info
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server ports = plugin.get_ports(context, filters=filters)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 233, in wrapped
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 140, in wrapped
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server self.force_reraise()
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server raise value
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 136, in wrapped
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 154, in wrapper
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server self.force_reraise()
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server raise value
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 142, in wrapper
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server self.force_reraise()
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server raise value
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1503, in get_ports
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server items = [self._make_port_dict(c, fields) for c in query]
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1503, in <listcomp>
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server items = [self._make_port_dict(c, fields) for c in query]
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/db/db_base_plugin_common.py", line 221, in _make_port_dict
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server resource_extend.apply_funcs(port_def.COLLECTION_NAME, res, port)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/resource_extend.py", line 84, in apply_funcs
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server resolved_func(response, db_object)
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server File "/opt/stack/neutron/neutron/services/qos/qos_plugin.py", line 102, in _extend_port_resource_request
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server if net.qos_policy_id:
Jan 11 01:12:04.832940 ubuntu-bionic-ovh-bhs1-0001629903 neutron-server[15297]: ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'qos_policy_id'

It appears that the qos_plugin is always assuming it will get a network object back for ports:
neutron/services/qos/qos_plugin.py: L97

        # Note(lajoskatona): handle the case when the port inherits qos-policy
        # from the network.
        if not qos_policy:
            net = network_object.Network.get_object(
                context.get_admin_context(), id=port_res['network_id'])
            if net.qos_policy_id:
                qos_policy = policy_object.QosPolicy.get_network_policy(
                    context.get_admin_context(), net.id)

I think this needs to be updated to handle the case that a network is not returned.

Tags: qos
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/630361

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Michael Johnson (<email address hidden>) on branch: master
Review: https://review.openstack.org/630361
Reason: Duplicate to another patch:
https://review.openstack.org/#/c/628492/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.