Removing of QoS queue in neutron-ovs-agent fails due to existing references

Bug #1845176 reported by Slawek Kaplonski on 2019-09-24
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Medium
Rodolfo Alonso

Bug Description

It happend in fullstack test: neutron.tests.fullstack.test_qos.TestMinBwQoSOvs.test_bw_limit_qos_port_removed(egress). Test failed and in neutron-openvswitch-agent logs we found error like:

2019-09-17 05:35:50.732 14622 DEBUG neutron.agent.common.ovs_lib [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Processing 1 OpenFlow rules. do_action_flows /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py:432
2019-09-17 05:35:50.733 14622 DEBUG neutron.agent.linux.utils [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running command (rootwrap daemon): ['ovs-ofctl', 'del-flows', '-O', 'OpenFlow10', 'br-intad6ecf25c', '-'] execute_rootwrap_daemon /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py:103
2019-09-17 05:35:50.750 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbClearCommand(table=QoS, record=0e930eb1-49eb-4f9e-8639-267abba5f4aa, column=queues) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-09-17 05:35:50.757 14622 DEBUG ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Running txn n=1 command(idx=0): DbDestroyCommand(table=Queue, record=a1470780-1834-48d9-afd0-6fe41fcbb027) do_commit /home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:84
2019-09-17 05:35:50.762 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.764 14622 ERROR ovsdbapp.backend.ovs_idl.transaction [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
    txn.results.put(txn.do_commit())
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
    raise RuntimeError(msg)
RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}

2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error executing command: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command yield self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command KeyError: 140632266952480
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command During handling of the above exception, another exception occurred:
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command t.add(self)
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command next(self.gen)
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command del self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit()
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command raise result.ex
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit())
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg)
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.765 14622 ERROR ovsdbapp.backend.ovs_idl.command
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-afe98974-4696-4632-9f6c-a5f19fee65cd - - - - -] Error while processing VIF ports: RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 111, in transaction
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent yield self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 140632266952480
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2435, in rpc_loop
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1968, in process_network_ports
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent devices_added_updated, provisioning_needed))
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1834, in treat_devices_added_or_updated
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.ext_manager.delete_port(self.context, {'port_id': device})
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/l2_agent_extensions_manager.py", line 54, in delete_port
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent extension.obj.delete_port(context, data)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 279, in delete_port
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._process_reset_port(port)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 299, in _process_reset_port
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.qos_driver.delete(port)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 94, in delete
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._handle_rule_delete(port, rule_type)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/l2/extensions/qos.py", line 119, in _handle_rule_delete
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent handler(port)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/extension_drivers/qos_driver.py", line 218, in delete_minimum_bandwidth
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.br_int.delete_minimum_bandwidth_queue(port['port_id'])
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 985, in delete_minimum_bandwidth_queue
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._delete_queue(queue['_uuid'])
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/common/ovs_lib.py", line 1057, in _delete_queue
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.ovsdb.db_destroy('Queue', queue_id).execute(check_error=True)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent t.add(self)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent next(self.gen)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent del self._nested_txns_map[cur_thread_id]
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.result = self.commit()
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise result.ex
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent txn.results.put(txn.do_commit())
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 115, in do_commit
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise RuntimeError(msg)
2019-09-17 05:35:50.769 14622 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RuntimeError: OVSDB Error: {"details":"cannot delete Queue row a1470780-1834-48d9-afd0-6fe41fcbb027 because of 1 remaining reference(s)","error":"referential integrity violation"}

Full log can be found at https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_847/681893/1/gate/neutron-fullstack/847f4d9/controller/logs/dsvm-fullstack-logs/TestMinBwQoSOvs.test_bw_limit_qos_port_removed_egress_/neutron-openvswitch-agent--2019-09-17--05-35-29-575569_log.txt.gz

Changed in neutron:
assignee: nobody → Rodolfo Alonso (rodolfo-alonso-hernandez)
status: Confirmed → In Progress
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers