neutron

LBaaS: stacktraces in q-svc while running tempest lbaas api tests

Bug #1260682 reported by Oleg Bondarev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Oleg Bondarev
neutron 2014.1 "icehouse"

Bug Description

Followin stacktraces appeared after merging https://review.openstack.org/#/c/40381:

2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp **args)
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/common/rpc.py", line 45, in dispatch
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp neutron_ctxt, version, method, namespace, **kwargs)
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/plugin_driver.py", line 171, in update_status
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp context, obj_id['monitor_id'], obj_id['pool_id'], status)
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/db/loadbalancer/loadbalancer_db.py", line 651, in update_pool_health_monitor
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp assoc = self._get_pool_health_monitor(context, id, pool_id)
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/db/loadbalancer/loadbalancer_db.py", line 635, in _get_pool_health_monitor
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp monitor_id=id, pool_id=pool_id)
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp PoolMonitorAssociationNotFound: Monitor 7cea505d-d5cb-4d3f-958a-6dd14edb65e1 is not associated with Pool 2ce72926-fcbb-442e-b9e6-7724ec6c472c
2013-12-12 14:34:39.961 6522 TRACE neutron.openstack.common.rpc.amqp
2013-12-12 14:34:39.963 6522 ERROR neutron.openstack.common.rpc.common [req-bc48bc7a-b9dd-429f-a522-ab6ff56adfc9 None None] Returning exception Monitor 7cea505d-d5cb-4d3f-958a-6dd14edb65e1 is not associated with Pool 2ce72926-fcbb-442e-b9e6-7724ec6c472c to caller
2013-12-12 14:34:39.963 6522 ERROR neutron.openstack.common.rpc.common [req-bc48bc7a-b9dd-429f-a522-ab6ff56adfc9 None None] ['Traceback (most recent call last):\n', ' File "/opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data\n **args)\n', ' File "/opt/stack/new/neutron/neutron/common/rpc.py", line 45, in dispatch\n neutron_ctxt, version, method, namespace, **kwargs)\n', ' File "/opt/stack/new/neutron/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n result = getattr(proxyobj, method)(ctxt, **kwargs)\n', ' File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/plugin_driver.py", line 171, in update_status\n context, obj_id[\'monitor_id\'], obj_id[\'pool_id\'], status)\n', ' File "/opt/stack/new/neutron/neutron/db/loadbalancer/loadbalancer_db.py", line 651, in update_pool_health_monitor\n assoc = self._get_pool_health_monitor(context, id, pool_id)\n', ' File "/opt/stack/new/neutron/neutron/db/loadbalancer/loadbalancer_db.py", line 635, in _get_pool_health_monitor\n monitor_id=id, pool_id=pool_id)\n', 'PoolMonitorAssociationNotFound: Monitor 7cea505d-d5cb-4d3f-958a-6dd14edb65e1 is not associated with Pool 2ce72926-fcbb-442e-b9e6-7724ec6c472c\n']
2013-12-12 14:34:40.265 6522 ERROR neutron.openstack.common.rpc.amqp [-] Exception during message handling
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp Traceback (most recent call last):
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/openstack/common/rpc/amqp.py", line 438, in _process_data
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp **args)
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/common/rpc.py", line 45, in dispatch
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp neutron_ctxt, version, method, namespace, **kwargs)
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/plugin_driver.py", line 174, in update_status
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp context, model_mapping[obj_type], obj_id, status)
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/db/loadbalancer/loadbalancer_db.py", line 194, in update_status
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp v_db = self._get_resource(context, model, id)
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp File "/opt/stack/new/neutron/neutron/db/loadbalancer/loadbalancer_db.py", line 212, in _get_resource
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp raise loadbalancer.MemberNotFound(member_id=id)
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp MemberNotFound: Member 21f6ce73-fff8-4a7e-b798-527a4bb64b92 could not be found
2013-12-12 14:34:40.265 6522 TRACE neutron.openstack.common.rpc.amqp

Tags: lbaas
Revision history for this message
Oleg Bondarev (obondarev) wrote :

Stacktraces are caused by agent sending update_status requests to server on objects which were already deleted from db: this is due to API tests creating and deleting objects with a high rate. The resolution would be to catch NotFound exceptions and print a warning.

Changed in neutron:
milestone: none → icehouse-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/62318

Changed in neutron:
status: New → In Progress
Thierry Carrez (ttx)
Changed in neutron:
milestone: icehouse-2 → icehouse-3
Oleg Bondarev (obondarev)
Changed in neutron:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.