UnboundLocalError: local variable 'network_name' referenced before assignment

Bug #1246258 reported by Gary Kotton
136
This bug affects 23 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Gary Kotton
Havana
Fix Released
Undecided
wingwj

Bug Description

The exception occurs when trying to create/delete an instance that is using a network that is not owned by the admin tenant. This prevents the deletion of the instance.

2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp payload)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp pass
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1616, in run_instance
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp do_run_instance()
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1615, in do_run_instance
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp legacy_bdm_in_spec)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 965, in _run_instance
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp notify("error", msg=unicode(e)) # notify that build failed
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 949, in _run_instance
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp instance, image_meta, legacy_bdm_in_spec)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1078, in _build_instance
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp filter_properties, bdms, legacy_bdm_in_spec)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1122, in _reschedule_or_error
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp self._log_original_error(exc_info, instance_uuid)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1117, in _reschedule_or_error
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp bdms, requested_networks)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1642, in _shutdown_instance
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp network_info = self._get_instance_nw_info(context, instance)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 879, in _get_instance_nw_info
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp instance)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 455, in get_instance_nw_info
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp result = self._get_instance_nw_info(context, instance, networks)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 463, in _get_instance_nw_info
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp nw_info = self._build_network_info_model(context, instance, networks)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 1009, in _build_network_info_model
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp subnets)
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/neutronv2/api.py", line 962, in _nw_info_build_network
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp label=network_name,
2013-10-29 00:51:36.173 18588 TRACE nova.openstack.common.rpc.amqp UnboundLocalError: local variable 'network_name' referenced before assignment

Revision history for this message
Gary Kotton (garyk) wrote :
tags: added: havana-backport-potential
Changed in nova:
importance: Undecided → High
assignee: nobody → Gary Kotton (garyk)
milestone: none → icehouse-1
Changed in nova:
status: New → In Progress
Revision history for this message
Sheldon Hearn (sheldonh) wrote :

Workaround?

Revision history for this message
Sheldon Hearn (sheldonh) wrote :

This trap is really easy to fall into, because neutron net-create doesn't seem to validate the --tenant-id argument:

# neutron net-create --tenant-id admin demo-net
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 427d4260-b246-4789-b543-58ce1e86b51c |
| name | demo-net |
| provider:network_type | gre |
| provider:physical_network | |
| provider:segmentation_id | 1 |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | admin |
+---------------------------+--------------------------------------+

So you end up with a poison network that breaks the metadata service. Recovery isn't an obvious process. You have to start by deleting the ports that connect affected instances into the poison network. Then you can delete the subnet and finally the network.

Revision history for this message
Sheldon Hearn (sheldonh) wrote :

Does this bug manifest when the instance is connected to a network in a correctly identified tenant that is not the admin tenant, or is it only when the network is not in a correctly identified tenant (as per comment #3)?

Changed in nova:
milestone: icehouse-1 → icehouse-2
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-2 → icehouse-3
Revision history for this message
George Shuklin (george-shuklin) wrote :

Here simple patch which allow to remove instances with incorrect network interconnection.

Problem splited into two pieces:
1) Disallow using other tenant's networks for different tenant's instances.
2) Allow gracefully remove them.

This patch fix only 2nd problem.

Revision history for this message
George Shuklin (george-shuklin) wrote :

... and I think this bug should be backported to havanna, because any guy who play with API can stuck with undeletable instances.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/54521
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b12da559b6a40fba4e4431d74372b5e350047525
Submitter: Jenkins
Branch: master

commit b12da559b6a40fba4e4431d74372b5e350047525
Author: Gary Kotton <email address hidden>
Date: Wed Oct 30 04:21:53 2013 -0700

    Fix bug for neutron network-name

    There may be cases when the network-name is not updated correctly
    due to the requested networks not being owned by the tenant.

    In the case when there is no matched network we use the details
    from the used port.

    A failed deletion of a port will raise an exception. If the
    port is not found no exception will be raised.

    Co-authored-by: Evgeny Fedoruk <email address hidden>

    Change-Id: Ie28e88ec9a9c7a180410ae5e57f84b1f653bbffc
    Closes-Bug: #1246258

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/69289

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/havana)

Reviewed: https://review.openstack.org/69289
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5b0a13146d736adf5ca70649c7e17037b14a3d7d
Submitter: Jenkins
Branch: stable/havana

commit 5b0a13146d736adf5ca70649c7e17037b14a3d7d
Author: Gary Kotton <email address hidden>
Date: Wed Oct 30 04:21:53 2013 -0700

    Fix bug for neutron network-name

    There may be cases when the network-name is not updated correctly
    due to the requested networks not being owned by the tenant.

    In the case when there is no matched network we use the details
    from the used port.

    A failed deletion of a port will raise an exception. If the
    port is not found no exception will be raised.

    Co-authored-by: Evgeny Fedoruk <email address hidden>

    Closes-Bug: #1246258
    (cherry picked from commit b12da559b6a40fba4e4431d74372b5e350047525)

    Conflicts:
     nova/tests/network/test_neutronv2.py

    Change-Id: Ie28e88ec9a9c7a180410ae5e57f84b1f653bbffc

tags: added: in-stable-havana
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-3 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.