ovs agent get exception when processing VIF ports

[yalei wang]

Mark this error here, not sure if it is a neutron's bug

=============================
2015-01-22 15:46:12.460 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-bde320a0-61da-4f4b-8710-eef1e8c8819b None None] Error while processing VIF ports
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1455, in rpc_loop
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent ovs_restarted)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1221, in process_network_ports
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 342, in setup_port_filters
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 201, in decorated_function
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/agent/securitygroups_rpc.py", line 226, in prepare_devices_filter
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent security_groups, security_group_member_ips)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.gen.next()
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/agent/firewall.py", line 106, in defer_apply
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.filter_defer_apply_off()
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/agent/linux/iptables_firewall.py", line 539, in filter_defer_apply_off
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.iptables.defer_apply_off()
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 397, in defer_apply_off
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self._apply()
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 412, in _apply
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent with lockutils.lock(lock_name, utils.SYNCHRONIZED_PREFIX, True):
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return self.gen.next()
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 381, in lock
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent ext_lock = external_lock(name, lock_file_prefix, lock_path)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 318, in external_lock
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent lock_file_path = _get_lock_path(name, lock_file_prefix, lock_path)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 309, in _get_lock_path
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent local_lock_path = lock_path or CONF.oslo_concurrency.lock_path
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2384, in __getattr__
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return self._conf._get(name, self._group)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2116, in _get
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent value = self._do_get(name, group, namespace)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2153, in _do_get
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return convert(opt._get_from_namespace(namespace, group_name))
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2148, in convert
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self._substitute(value, group, namespace), opt)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2199, in _substitute
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self.StrSubWrapper(self, group=group, namespace=namespace))
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/string.py", line 205, in safe_substitute
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return self.pattern.sub(convert, self.template)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/string.py", line 190, in convert
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent return '%s' % (mapping[named],)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2464, in __getitem__
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent value = self.conf._get(key, namespace=self.namespace)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2116, in _get
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent value = self._do_get(name, group, namespace)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2134, in _do_get
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent info = self._get_opt_info(name, group)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py", line 2262, in _get_opt_info
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent raise NoSuchOptError(opt_name, group)
2015-01-22 15:46:12.460 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent NoSuchOptError: no such option: state_path

[yalei wang (yalei-wang)]

no such issure before commit 35d838ff600729f4b7b2ac8be92d3a3be4d96cb8

[Darragh O'Reilly (darragh-oreilly)]

I saw something like this after git pulling. I had to:

$ sudo pip install --upgrade oslo.concurrency

So maybe the minimum requirement needs to be bumped or something

$ grep oslo.concurrency /opt/stack/neutron/requirements.txt
oslo.concurrency>=0.3.0,!=0.4.0

[yalei wang (yalei-wang)]

the reason for this bug may mostly related to the l2agt and l3agt not register the state_path, like lock_path in neutron.conf

[yalei wang (yalei-wang)]

because /lockutils.py changed from neutron/openstack/common to oslo_concurrency

the lock_path could not resolved in CONF.oslo_concurrency.lock_path with $state_path

then I think we have 3 choice:

1. edit devstack to set the ini with full path.

2. remove the $state_path using in etc/neutron.conf

3. make the oslo_concurrency support $state_path (doubt about it)

[yalei wang (yalei-wang)]

Hi Darragh O'Reilly, the workaround is use full path like this for lock_path

lock_path = /opt/stack/data/neutron/lock

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/149514

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/149521

[OpenStack Infra (hudson-openstack)]

Change abandoned by yalei wang (<email address hidden>) on branch: master
Review: https://review.openstack.org/149521
Reason: this patch would fix it
https://review.openstack.org/#/c/149400

[OpenStack Infra (hudson-openstack)]

Change abandoned by yalei wang (<email address hidden>) on branch: master
Review: https://review.openstack.org/149514
Reason: a better fix
https://review.openstack.org/149400

[Isaku Yamahata (yamahata)]

It seems state_path comes from lock_path=$state_path/lock.
So it seems registration of state_path to oslo.config was dropped somehow.

[George (lmihaiescu)]

Any chance this can be backported to Juno?