neutron

LBaaS: stacktraces in q-lbaas while running tempest lbaas api tests

Bug #1260692 reported by Oleg Bondarev
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Oleg Bondarev
neutron 2014.1 "icehouse"

Bug Description

Following stacktraces appeared in q-lbaas after merging https://review.openstack.org/#/c/40381:

2013-12-10 14:02:10.655 6158 ERROR neutron.services.loadbalancer.drivers.haproxy.agent_manager [req-0316f127-ac85-4d14-b6c6-57d58272f9e6 c7798670e00d40f7809a65f04536b969 f21028acd29f43dc935c5aeb3950bb06] Create member b4b0bd55-a70e-48f5-8816-fdbcba372cc1 failed on device driver haproxy_ns
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Traceback (most recent call last):
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/agent_manager.py", line 276, in create_member
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager driver.create_member(member)
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 308, in create_member
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager self._refresh_device(member['pool_id'])
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 283, in _refresh_device
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager self.deploy_instance(logical_config)
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 279, in deploy_instance
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager self.create(logical_config)
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 90, in create
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager self._plug(namespace, logical_config['vip']['port'])
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 246, in _plug
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager namespace=namespace
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/interface.py", line 186, in plug
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager ns_dev.link.set_address(mac_address)
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 230, in set_address
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager self._as_root('set', self.name, 'address', mac_address)
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 217, in _as_root
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager kwargs.get('use_root_namespace', False))
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 70, in _as_root
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager namespace)
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 81, in _execute
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager root_helper=root_helper)
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 75, in execute
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager raise RuntimeError(m)
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager RuntimeError:
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'set', 'tapc8872a60-80', 'address', 'fa:16:3e:fe:37:64']
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Exit code: 1
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Stdout: ''
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Stderr: 'Cannot find device "tapc8872a60-80"\n'
2013-12-10 14:02:10.655 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager
2013-12-10 14:02:12.330 6158 ERROR neutron.services.loadbalancer.drivers.haproxy.agent_manager [req-832c5fe5-d69d-47af-bd09-0632d591ce62 c7798670e00d40f7809a65f04536b969 f21028acd29f43dc935c5aeb3950bb06] Create health_monitor {'monitor_id': u'540e405d-8d64-45fb-8494-da4c301923c9', 'pool_id': u'4c4dc051-4d2d-414e-8c7f-7ba281878509'} failed on device driver haproxy_ns
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Traceback (most recent call last):
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/agent_manager.py", line 303, in create_pool_health_monitor
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager driver.create_pool_health_monitor(health_monitor, pool_id)
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 317, in create_pool_health_monitor
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager self._refresh_device(pool_id)
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 283, in _refresh_device
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager self.deploy_instance(logical_config)
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 279, in deploy_instance
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager self.create(logical_config)
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 90, in create
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager self._plug(namespace, logical_config['vip']['port'])
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/services/loadbalancer/drivers/haproxy/namespace_driver.py", line 246, in _plug
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager namespace=namespace
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/interface.py", line 195, in plug
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager namespace_obj = ip.ensure_namespace(namespace)
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 136, in ensure_namespace
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager ip = self.netns.add(name)
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 447, in add
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager self._as_root('add', name, use_root_namespace=True)
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 217, in _as_root
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager kwargs.get('use_root_namespace', False))
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 70, in _as_root
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager namespace)
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 81, in _execute
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager root_helper=root_helper)
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 75, in execute
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager raise RuntimeError(m)
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager RuntimeError:
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'add', 'qlbaas-4c4dc051-4d2d-414e-8c7f-7ba281878509']
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Exit code: 1
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Stdout: ''
2013-12-10 14:02:12.330 6158 TRACE neutron.services.loadbalancer.drivers.haproxy.agent_manager Stderr: 'Could not create /var/run/netns/qlbaas-4c4dc051-4d2d-414e-8c7f-7ba281878509: File exists\n'

Tags: lbaas
Revision history for this message
Oleg Bondarev (obondarev) wrote :

tempest api tests do creating/deleting lbaas objects at a high rate so that deploy/undeploy_unstance methods of haproxy namespace driver are executed concurrently - this leads to different conflicts with network namespaces and interfaces created by the driver.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/62010

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/62010
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6d3290ab39db687d34c02caebfbb3d32c90e9cdf
Submitter: Jenkins
Branch: master

commit 6d3290ab39db687d34c02caebfbb3d32c90e9cdf
Author: Oleg Bondarev <email address hidden>
Date: Fri Dec 13 18:22:32 2013 +0400

    LBaaS: synchronize haproxy deploy/undeploy_instance methods

    When creating/deleting lbaas objects at a high rate (like tempest api tests do)
    deploy/undeploy_unstance methods of haproxy namespace driver
    may be executed concurrently which leads to different conflicts with
    network namespaces and interfaces created by the diver.

    @synchronized decorator should be used in order to ensure that only one thread
    will execute one of the methods at a time.

    Closes-Bug: #1260692

    Change-Id: Ibe86653c20c9a4e0b4e7b3b2cefbd445428b82da

Changed in neutron:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in neutron:
milestone: none → icehouse-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: icehouse-2 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.