OpenStack Compute (nova)

nova interface-attach fails with quantum

Bug #1171636 reported by Aaron Rosen
28
This bug affects 6 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Dan Smith
OpenStack Compute (nova) 2013.2 "havana"
Grizzly
Fix Released
Medium
Russell Bryant
OpenStack Compute (nova) 2013.1.2

Bug Description

2013-04-22 15:12:07 ERROR [nova.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 433, in _process_data
    **args)
  File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 148, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/opt/stack/nova/nova/compute/manager.py", line 3046, in attach_interface
    self.conductor_api)
  File "/opt/stack/nova/nova/network/api.py", line 47, in wrapper
    res = f(self, context, *args, **kwargs)
  File "/opt/stack/nova/nova/network/quantumv2/api.py", line 342, in allocate_port_for_instance
    conductor_api=conductor_api)
  File "/opt/stack/nova/nova/network/api.py", line 47, in wrapper
    res = f(self, context, *args, **kwargs)
  File "/opt/stack/nova/nova/network/quantumv2/api.py", line 286, in allocate_for_instance
    nw_info = self._get_instance_nw_info(context, instance, networks=nets)
  File "/opt/stack/nova/nova/network/quantumv2/api.py", line 383, in _get_instance_nw_info
    nw_info = self._build_network_info_model(context, instance, networks)
  File "/opt/stack/nova/nova/network/quantumv2/api.py", line 827, in _build_network_info_model
    [n['id'] for n in networks])
  File "/opt/stack/nova/nova/network/quantumv2/api.py", line 984, in _ensure_requested_network_ordering
    unordered.sort(key=lambda i: preferred.index(accessor(i)))
  File "/opt/stack/nova/nova/network/quantumv2/api.py", line 984, in <lambda>
    unordered.sort(key=lambda i: preferred.index(accessor(i)))
ValueError: u'474719de-5160-48a9-8741-dcf3ff859978' is not in list
2013-04-22 15:12:07 ERROR [nova.openstack.common.rpc.common] Returning exception u'474719de-5160-48a9-8741-dcf3ff859978' is not in list to caller
2013-04-22 15:12:07 ERROR [nova.openstack.common.rpc.common] ['Traceback (most recent call last):\n', ' File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 433, in _process_data\n **args)\n', ' File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 148, in dispatch\n return getattr(proxyobj, method)(ctxt, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 3046, in attach_interface\n self.conductor_api)\n', ' File "/opt/stack/nova/nova/network/api.py", line 47, in wrapper\n res = f(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/quantumv2/api.py", line 342, in allocate_port_for_instance\n conductor_api=conductor_api)\n', ' File "/opt/stack/nova/nova/network/api.py", line 47, in wrapper\n res = f(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/network/quantumv2/api.py", line 286, in allocate_for_instance\n nw_info = self._get_instance_nw_info(context, instance, networks=nets)\n', ' File "/opt/stack/nova/nova/network/quantumv2/api.py", line 383, in _get_instance_nw_info\n nw_info = self._build_network_info_model(context, instance, networks)\n', ' File "/opt/stack/nova/nova/network/quantumv2/api.py", line 827, in _build_network_info_model\n [n[\'id\'] for n in networks])\n', ' File "/opt/stack/nova/nova/network/quantumv2/api.py", line 984, in _ensure_requested_network_ordering\n unordered.sort(key=lambda i: preferred.index(accessor(i)))\n', ' File "/opt/stack/nova/nova/network/quantumv2/api.py", line 984, in <lambda>\n unordered.sort(key=lambda i: preferred.index(accessor(i)))\n', "ValueError: u'474719de-5160-48a9-8741-dcf3ff859978' is not in list\n"]

Aaron Rosen (arosen)
Changed in nova:
assignee: nobody → Aaron Rosen (arosen)
Revision history for this message
Aaron Rosen (arosen) wrote :

unassigned for now... will pick up later if no one else does. I have some other stuff on my plate I need to fix first.

Changed in nova:
assignee: Aaron Rosen (arosen) → nobody
Revision history for this message
Aaron Rosen (arosen) wrote :

To reproduce:

create two networks:

quantum net-create net1
quantum net-create net2
quantum subnet-create net1 10.0.0.0/24
quantum subnet-create net2 11.0.0.0/24

nova boot --nic net-id=<net-1-uuid> .... vm1
nova interface-attach --net-id <net2-uuid> <vm1>

Dan Smith (danms)
Changed in nova:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/27820

Changed in nova:
assignee: nobody → Dan Smith (danms)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/27821

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/27820
Committed: http://github.com/openstack/nova/commit/08c7cf5c67a75dea9b453bd915a0cd55e67c4adc
Submitter: Jenkins
Branch: master

commit 08c7cf5c67a75dea9b453bd915a0cd55e67c4adc
Author: Dan Smith <email address hidden>
Date: Mon Apr 29 16:33:49 2013 -0700

    Fix building quantumapi network model with network list

    Bug 1171636 occurs when _build_network_info_model() receives a list
    of networks that does not fully map all of an instance's ports.
    It crashes in _ensure_requested_network_ordering() when the list of
    all ports is sorted by the subset of networks provided.

    This change simplifies the early part of that function to only
    consider ports within the networks provided, and to always sort those
    ports in network order. This allows us to remove some error path
    code from the loop below that iterates over the ports (which contained
    a fatal typo, by the way) which is now dead code.

    The affected code is really untested at the moment, due to the
    structure of the method, making it hard to add a unit test to cover
    this fix. In order to make the change clear from the existing code,
    this fix is provided here first, and the following patch restructures
    the method and add tests for this change and the rest of the surrounding
    code.

    Fixes bug 1171636

    Change-Id: Idf519a88415844451b18959b38cc48ce92c090d1

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/27821
Committed: http://github.com/openstack/nova/commit/43d657a11e953660f36683d595976c8bd8edb4cc
Submitter: Jenkins
Branch: master

commit 43d657a11e953660f36683d595976c8bd8edb4cc
Author: Dan Smith <email address hidden>
Date: Tue Apr 30 09:11:48 2013 -0700

    Make _build_network_info_model testable

    This restructures the quantumapi _build_network_info_model()
    method to be testable and adds tests for each of the smaller
    components. It also provides verification for bug 1171636 by
    ensuring that ports returned from the client that are not in the
    set of networks passed don't cause the failure detailed in that
    bug.

    Change-Id: I82f20f68122fc38e72c31e52283529614252c174

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/28077

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/28078

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/grizzly)

Reviewed: https://review.openstack.org/28077
Committed: http://github.com/openstack/nova/commit/bba57e9fa63b7f55d403d9f6950c4cde425d83b0
Submitter: Jenkins
Branch: stable/grizzly

commit bba57e9fa63b7f55d403d9f6950c4cde425d83b0
Author: Dan Smith <email address hidden>
Date: Mon Apr 29 16:33:49 2013 -0700

    Fix building quantumapi network model with network list

    Bug 1171636 occurs when _build_network_info_model() receives a list
    of networks that does not fully map all of an instance's ports.
    It crashes in _ensure_requested_network_ordering() when the list of
    all ports is sorted by the subset of networks provided.

    This change simplifies the early part of that function to only
    consider ports within the networks provided, and to always sort those
    ports in network order. This allows us to remove some error path
    code from the loop below that iterates over the ports (which contained
    a fatal typo, by the way) which is now dead code.

    The affected code is really untested at the moment, due to the
    structure of the method, making it hard to add a unit test to cover
    this fix. In order to make the change clear from the existing code,
    this fix is provided here first, and the following patch restructures
    the method and add tests for this change and the rest of the surrounding
    code.

    Fixes bug 1171636

    Change-Id: Idf519a88415844451b18959b38cc48ce92c090d1
    (cherry picked from commit 08c7cf5c67a75dea9b453bd915a0cd55e67c4adc)

tags: added: in-stable-grizzly
Thierry Carrez (ttx)
Changed in nova:
milestone: none → havana-1
status: Fix Committed → Fix Released
Alan Pevec (apevec)
tags: removed: in-stable-grizzly
Revision history for this message
Naga Praveen Illa (inp-bitsg) wrote :
Download full text (5.7 KiB)

I see that the bug https://bugs.launchpad.net/nova/+bug/1171636

is still reproducible when I follow the steps (im using stable/grizzly branch)

The error log is as below:
——————————-
x-image-meta-size: 25165824
x-image-meta-is_public: True
content-type: text/html; charset=UTF-8
x-image-meta-disk_format: ami
log_http_response /opt/stack/python-glanceclient/glanceclient/common/http.py:150
2013-08-28 17:53:30.068 DEBUG nova.openstack.common.rpc.amqp [req-a4ad554e-ab0e-4864-8eb6-a086eebe9021 demo demo] Making synchronous call on conductor … multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:513
2013-08-28 17:53:30.069 DEBUG nova.openstack.common.rpc.amqp [req-a4ad554e-ab0e-4864-8eb6-a086eebe9021 demo demo] MSG_ID is 5624c471e3604533bfe29f0b9e33e5be multicall /opt/stack/nova/nova/openstack/common/rpc/amqp.py:516
2013-08-28 17:53:30.069 DEBUG nova.openstack.common.rpc.amqp [req-a4ad554e-ab0e-4864-8eb6-a086eebe9021 demo demo] UNIQUE_ID is e417b9a25b934feb85de7e41fbad6c06. _add_unique_id /opt/stack/nova/nova/openstack/common/rpc/amqp.py:322
2013-08-28 17:53:30.071 25065 DEBUG amqp [-] Closed channel #1 _do_close /usr/local/lib/python2.7/dist-packages/amqp/channel.py:88
2013-08-28 17:53:30.071 25065 DEBUG amqp [-] using channel_id: 1 __init__ /usr/local/lib/python2.7/dist-packages/amqp/channel.py:70
2013-08-28 17:53:30.072 25065 DEBUG amqp [-] Channel open _open_ok /usr/local/lib/python2.7/dist-packages/amqp/channel.py:420
2013-08-28 17:53:30.096 ERROR nova.openstack.common.rpc.amqp [req-a4ad554e-ab0e-4864-8eb6-a086eebe9021 demo demo] Exception during message handling
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp File “/opt/stack/nova/nova/openstack/common/rpc/amqp.py”, line 421, in _process_data
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp **args)
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp File “/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py”, line 172, in dispatch
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp File “/opt/stack/nova/nova/compute/manager.py”, line 3642, in attach_interface
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp self.driver.attach_interface(instance, image_meta, network_info[0])
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp File “/opt/stack/nova/nova/virt/libvirt/driver.py”, line 1174, in attach_interface
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp self.vif_driver.plug(instance, vif)
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp File “/opt/stack/nova/nova/virt/libvirt/vif.py”, line 750, in plug
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp return self.plug_ovs_hybrid(instance, vif)
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp File “/opt/stack/nova/nova/virt/libvirt/vif.py”, line 400, in plug_ovs_hybrid
2013-08-28 17:53:30.096 25065 TRACE nova.openstack.common.rpc.amqp iface_id = se...

Read more...

Thierry Carrez (ttx)
Changed in nova:
milestone: havana-1 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.