neutron

Removing of QoS queue in neutron-ovs-agent fails due to existing references

Bug #1845176 reported by Slawek Kaplonski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Rodolfo Alonso

Bug Description

It happend in fullstack test: neutron.tests.fullstack.test_qos.TestMinBwQoSOvs.test_bw_limit_qos_port_removed(egress). Test failed and in neutron-openvswitch-agent logs we found error like:

2019-09-17 05:35:50.732 14622 DEBUG neutron.agent.common.ovs_lib [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Processing 1 OpenFlow rules. do_action_flows /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py:432
2019-09-17 05:35:50.733 14622 DEBUG neutron.agent.linux.utils [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running command (rootwrap daemon): ['ovs-ofctl', 'del-flows', '-O', 'OpenFlow10', 'br-intad6ecf25c', '-'] execute_rootwrap_daemon /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py:103
2019-09-17 05:35:50.750 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbClearCommand(table=QoS, record=0e930eb1-49eb-4f9e-8639-267abba5f4aa, column=queues) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-09-17 05:35:50.757 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbDestroyCommand(table=Queue, record=a1470780-1834-48d9-afd0-6fe41fcbb027) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-09-17 05:35:50.762 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.764 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
    txn.results.put(txn.do_commit())
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
    raise RuntimeError(msg)
RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}

2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error executing command: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command yield self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command KeyError: 140632266952480
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command During handling of the above exception, another exception occurred:
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command t.add(self)
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command next(self.gen)
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command del self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit()
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command raise result.ex
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit())
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg)
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error while processing VIF ports: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent yield self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 140632266952480
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2435, in rpc_loop
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1968, in process_network_ports
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent devices_added_updated, provisioning_needed))
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1834, in treat_devices_added_or_updated
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.ext_manager.delete_port(self.context, {'port_id': device})
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/l2_agent_extensions_manager.py", line 54, in delete_port
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent extension.obj.delete_port(context, data)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 279, in delete_port
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._process_reset_port(port)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 299, in _process_reset_port
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.qos_driver.delete(port)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 94, in delete
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._handle_rule_delete(port, rule_type)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 119, in _handle_rule_delete
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent handler(port)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py", line 218, in delete_minimum_bandwidth
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.br_int.delete_minimum_bandwidth_queue(port['port_id'])
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 985, in delete_minimum_bandwidth_queue
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._delete_queue(queue['_uuid'])
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 1057, in _delete_queue
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.ovsdb.db_destroy('Queue', queue_id).execute(check_error=True)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent t.add(self)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent next(self.gen)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent del self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.result = self.commit()
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise result.ex
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent txn.results.put(txn.do_commit())
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise RuntimeError(msg)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}

Full log can be found at https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_847/681893/1/gate/neutron-fullstack/847f4d9/controller/logs/dsvm-fullstack-logs/TestMinBwQoSOvs.test_bw_limit_qos_port_removed_egress_/neutron-openvswitch-agent--2019-09-17--05-35-29-575569_log.txt.gz

OpenStack Infra (hudson-openstack)
Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
status: Confirmed → In Progress
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Patch: https://review.opendev.org/#/c/687922/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/687922
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=83d7eb961dc8a5d4e815bc3f110f8523a532caeb
Submitter: Zuul
Branch: master

commit 83d7eb961dc8a5d4e815bc3f110f8523a532caeb
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Oct 15 15:51:40 2019 +0000

    Add OVS QoS driver cache for minimum bandwidth rules

    1. Add OVS QoS driver cache
    Added minimum bandwidth rules cache in the OVS QoS driver. If a new
    port is detected in the integration bridge, managed by the OVS agent
    driver, the QoS extension is applied (if configured).

    If this new port does not have a QoS rule, by default the QoS
    configuration is reset in this port. Because this port has no Queue
    related, the call "QosOVSAgentDriver.delete_minimum_bandwidth" should
    not execute the rule cleanup.

    This QoS rule cache implementation per porti is currently used in the
    OVS QoS driver for the DSCP and the bandwitdh limit rules.

    2. If a Queue cannot be deleted, log the QoS registers using it
    If a Queue is still used in a QoS register in the OVSDB, it cannot
    be deleted. With the current design, only one QoS rule is created in
    the database and several Queues are assigned to it.

    If something external to the OVS agent is handling the Queues and by
    mistake a Queue is assigned to other QoS registers, this error is now
    logged before raising the exception.

    Closes-Bug: #1845176
    Change-Id: Ia9077fc20e4ca360819a2e368c8c1f9250e5a6d8

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.0.0.0b1

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.