local variable 'existing_floating_ips' referenced before assignment in file agent/l3/agent.py

Bug #1413111 reported by yalei wang
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Brian Haley

Bug Description

step:

1. create the the firewall when there is no outer in this tenant.
2. create a router
3. delete the firewall
4. restart the l3-agent.

an exception emit

=====================================

2015-01-21 15:32:25.173 ERROR neutron.agent.l3.agent [req-f53cd6a8-094d-45d9-968c-4db962d5ecb4 None 12558d21504946d0a4bc9994409118cd] local variable 'existing_floating_ips' referenced before assignment
2015-01-21 15:32:25.173 TRACE neutron.agent.l3.agent Traceback (most recent call last):
2015-01-21 15:32:25.173 TRACE neutron.agent.l3.agent File "/opt/stack/neutron/neutron/common/utils.py", line 341, in call
2015-01-21 15:32:25.173 TRACE neutron.agent.l3.agent return func(*args, **kwargs)
2015-01-21 15:32:25.173 TRACE neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 566, in process_router
2015-01-21 15:32:25.173 TRACE neutron.agent.l3.agent self._process_external(ri)
2015-01-21 15:32:25.173 TRACE neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 535, in _process_external
2015-01-21 15:32:25.173 TRACE neutron.agent.l3.agent try:
2015-01-21 15:32:25.173 TRACE neutron.agent.l3.agent UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment
2015-01-21 15:32:25.173 TRACE neutron.agent.l3.agent
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
    func(*args, **kwargs)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1075, in _process_router_update
    self._process_router_if_compatible(router)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1029, in _process_router_if_compatible
    self._process_added_router(router)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 1038, in _process_added_router
    self.process_router(ri)
  File "/opt/stack/neutron/neutron/common/utils.py", line 344, in call
    self.logger(e)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/neutron/neutron/common/utils.py", line 341, in call
    return func(*args, **kwargs)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 566, in process_router
    self._process_external(ri)
  File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 535, in _process_external
    try:
UnboundLocalError: local variable 'existing_floating_ips' referenced before assignment

Tags: l3-ipam-dhcp
Revision history for this message
yalei wang (yalei-wang) wrote :

after reviewing the code, defer_apply could return IpTablesApplyException, and that make _process_external in agent/l3/agent.py go directly into L554 except (n_exc.FloatingIpSetupException, n_exc.IpTablesApplyException):

then " local variable 'existing_floating_ips' referenced before assignment "

Changed in neutron:
assignee: nobody → yalei wang (yalei-wang)
yalei wang (yalei-wang)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/148862

Changed in neutron:
status: New → In Progress
Revision history for this message
Brian Haley (brian-haley) wrote :
Download full text (10.6 KiB)

I started debugging this by removing the exception code from the callers so I could see what the iptables manager was really triggering. It's a strange error to me at the moment:

2015-01-22 13:04:48.477 DEBUG oslo_concurrency.lockutils [req-d8769e5e-b77f-4524-947d-360da6508613 None None] Acquired semaphore "iptables-qrouter-4c732712-815b-4be0-a33b-2896e535141a" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:377
2015-01-22 13:04:48.477 DEBUG oslo_concurrency.lockutils [req-d8769e5e-b77f-4524-947d-360da6508613 None None] Releasing semaphore "iptables-qrouter-4c732712-815b-4be0-a33b-2896e535141a" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:390
2015-01-22 13:04:48.478 DEBUG neutron.agent.linux.iptables_manager [req-d8769e5e-b77f-4524-947d-360da6508613 None None] Semaphore / lock released "iptables-qrouter-4c732712-815b-4be0-a33b-2896e535141a" _apply /opt/stack/neutron/neutron/agent/linux/iptables_manager.py:416
2015-01-22 13:04:48.478 ERROR neutron.agent.l3.agent [req-d8769e5e-b77f-4524-947d-360da6508613 None None] no such option: state_path
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent Traceback (most recent call last):
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent File "/opt/stack/neutron/neutron/common/utils.py", line 342, in call
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent return func(*args, **kwargs)
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 593, in process_router
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent self._process_external(ri)
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 567, in _process_external
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent self._process_snat_dnat_for_fip(ri)
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent self.gen.throw(type, value, traceback)
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 386, in defer_apply
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent self.defer_apply_off()
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 398, in defer_apply_off
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent self._apply()
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 412, in _apply
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent with lockutils.lock(lock_name, utils.SYNCHRONIZED_PREFIX, True):
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent return self.gen.next()
2015-01-22 13:04:48.478 28841 TRACE neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_...

Revision history for this message
Brian Haley (brian-haley) wrote :

Sorry about that cut/paste, the line wrapping makes it a little hard to read.

Revision history for this message
Brian Haley (brian-haley) wrote :

I think the bug is being caused by code at the end of _handle_router_snat_rules(), it's calling:

    iptables_manager.apply()

But that will be one of two possible iptables tables depending on whether we're running with DVR enabled (I am).

In that case, the caller has defferred processing in:

  ri.iptables_manager

not:

  ri.snat_iptables_manager

Which is what the code will call in the DVR case.

I'll start working on a change for that and see how it goes.

Revision history for this message
Brian Haley (brian-haley) wrote :

Never mind my last comment, I actually traced this pack to a package problem - oslo.concurrenty 1.4.1 or greater is requred, else lockutils.lock() code will throw an exception trying to determine lock_path. By default, lock_patch = $state_path/lock in neutron.conf and that's causing trouble.

So this might actually be fixed with https://review.openstack.org/#/c/149400/

Revision history for this message
yalei wang (yalei-wang) wrote :

this is two defect, actually I filed another bug to trace the exception issue

https://bugs.launchpad.net/neutron/+bug/1413503

tags: added: l3-ipam-dhcp
Changed in neutron:
importance: Undecided → High
Changed in neutron:
assignee: yalei wang (yalei-wang) → Brian Haley (brian-haley)
Revision history for this message
YAMAMOTO Takashi (yamamoto) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/148862
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b117e8ec854dd0a26d04beb7e1effedffec3cb3a
Submitter: Jenkins
Branch: master

commit b117e8ec854dd0a26d04beb7e1effedffec3cb3a
Author: Yalei Wang <email address hidden>
Date: Wed Jan 21 17:48:56 2015 +0800

    Move the assignment of existing_floating_ips before try block

    In function _process_external() in agent/l3/agent.py, the call to
    iptables_manager.defer_apply() may throw an exception, making a
    later call to _update_fip_statuses() use an un-initialized value.
    This will throw its own UnboundLocalError, with the result being
    no iptables rules will be applied.

    Added tests to cover both the defer_apply() code exception
    processing, as well as this new case where we might jump to
    _update_fip_statuses() without having done any work on floating
    IP addresses.

    Change-Id: I0045effc9319772127758be4aacca02ab5c236cd
    Closes-Bug: #1413111

Changed in neutron:
status: In Progress → Fix Committed
Kyle Mestery (mestery)
Changed in neutron:
milestone: none → kilo-rc1
Thierry Carrez (ttx)
Changed in neutron:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in neutron:
milestone: kilo-rc1 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.