networking-cisco

Traceback ERROR: VNI delete validation failed during VM deletion

Bug #1417825 reported by Danny Choi on 2015-02-04

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	networking-cisco	New	Undecided	Unassigned

Bug Description

The following traceback is logged in screen-q-svc.log during VM deletion:

Traceback (most recent call last):
  File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 299, in _call_on_drivers
    getattr(driver.obj, method_name)(context)
  File "/opt/stack/neutron/neutron/openstack/common/lockutils.py", line 272, in inner
    return f(*args, **kwargs)
  File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py", line 416, in delete_port_postcommit
    self._delete_nve_member) if vxlan_segment else 0
  File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py", line 337, in _port_action_vxlan
    func(vni, device_id, mcast_group, host_id)
  File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py", line 160, in _delete_nve_member
    vni)
  File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 341, in delete_nve_member
    self._edit_config(nexus_host, config=confstr)
  File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 111, in _edit_config
    raise cexc.NexusConfigFailed(config=config, exc=e)
NexusConfigFailed: Failed to configure Nexus:
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
<interface>
    <nve>nve1</nve>
    <__XML__MODE_if-nve>
        <member>no member vni 9000</member>
    </__XML__MODE_if-nve>
</interface>

          </__XML__MODE__exec_configure>
        </configure>
      </config>
. Reason: ERROR: VNI delete validation failed
.
2015-02-03 15:51:35 ERROR thread=140215249078704 [neutron.plugins.ml2.plugin] mechanism_manager.delete_port_postcommit failed for port 4ad78d2a-c30f-4410-b8f8-d7c3ace379fd

Despite the error, the Nexus switch config is cleaned up correctly.

Steps to repro:
1. Pull in Rich's lockutil patch from staging/junoplus.
2. Using CLI, launch 100 VMs using the cirros image, each VM with 2 interfaces of 2 different networks.
3. Verify the N9Ks are configured correctly with the VXLAN parameters.
4. Check logs for error/traceback.
5. Note the errors in screen-n-cone.log:

screen-n-cond.log:2015-02-03 13:13:19.152 ERROR nova.scheduler.utils [req-6e3c07e8-2d6a-427b-8c27-858a7a821270 demo demo] [instance: a33d7012-3429-46f9-a160-3c863d2b74b2] Error from last host: qa5 (node qa5): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2033, in _do_build_and_run_instance\n filter_properties)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2164, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance a33d7012-3429-46f9-a160-3c863d2b74b2 was re-scheduled: Connection to neutron failed: HTTPConnectionPool(host='172.29.172.161', port=9696): Read timed out. (read timeout=30)\n"]
screen-n-cond.log:2015-02-03 13:13:20.050 ERROR nova.scheduler.utils [req-6e3c07e8-2d6a-427b-8c27-858a7a821270 demo demo] [instance: 9cd36f7f-13a5-41ea-84b2-3bc860ecb475] Error from last host: qa6 (node qa6): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2033, in _do_build_and_run_instance\n filter_properties)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2164, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance 9cd36f7f-13a5-41ea-84b2-3bc860ecb475 was re-scheduled: Connection to neutron failed: HTTPConnectionPool(host='172.29.172.161', port=9696): Read timed out. (read timeout=30)\n"]

6. Despite the errors, all 100 VMs come up active/running.
7. Using Horizon, delete 20 VMs at a time to delete all 100 VMs.
8. Verify the N9Ks config is removed correctly.
9. Check the traceback in screen-q-svc.log.
10. If no traceback, repeat steps 2 - 8.

Tags:

Revision history for this message

Danny Choi (dannchoi) wrote on 2015-02-04:

screen-q-svc.log Edit (7.5 MiB, text/html)

Revision history for this message

Danny Choi (dannchoi) wrote on 2015-02-10:

I would see traceback in screen-n-cone.log during creation, but no traceback in screen-q-svc.log during deletion in the first trial. Then in subsequent trials, traceback in screen-n-cond.log during creation, and traceback in screen-q-svc,log during deletion.

Despite the traceback, all VMs are deleted and created successfully in subsequent trials.

Revision history for this message

Danny Choi (dannchoi) wrote on 2015-03-05:

Download full text (5.5 KiB)

A similar traceback is logged during VM deletion in the following scenario:

1. Uses Carol's latest patch for replay code, https://review.gerrithub.io/217434
2. Using CLI, launch 100 VMs using the cirros image, each VM with 2 interfaces of 2 different networks, that span across 3 Compute nodes.
3. Verify the two N9Ks are configured correctly with the VXLAN parameters.
4. Verify all 100 VMs are active and running.
5. Reboot the N9K that has the Network node and Compute-1 connected to it.
6. Upon boot up, verify the replay code reconfigures the switch correctly with the VXLAN parameters.
7. Using Horizon, delete 20 VMs at a time to delete all 100 VMs.
8. Verify the N9Ks config is removed correctly.
9. Check the traceback in screen-q-svc.log.
10. Note the following traceback.

2015-03-05 10:33:03.610 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [req-27f75e6b-c7eb-4e8e-8979-804c2dc34d4d admin 05e1512cadd84617a8cb4bc1f3e64c80
] NexusDriver:
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
<interface>
    <nve>nve1</nve>
    <__XML__MODE_if-nve>
        <member>no member vni 222224</member>
    </__XML__MODE_if-nve>
</interface>

          </__XML__MODE__exec_configure>
        </configure>
      </config>
from (pid=31927) delete_nve_member /opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:380
2015-03-05 10:33:03.610 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [req-27f75e6b-c7eb-4e8e-8979-804c2dc34d4d admin 05e1512cadd84617a8cb4bc1f3e64c80
] NexusDriver config:
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
<interface>
    <nve>nve1</nve>
    <__XML__MODE_if-nve>
        <member>no member vni 222224</member>
    </__XML__MODE_if-nve>
</interface>

A similar traceback is logged during VM deletion in the following scenario:

2015-03-05 10:33:03.610 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [req-27f75e6b-c7eb-4e8e-8979-804c2dc34d4d admin 05e1512cadd84617a8cb4bc1f3e64c80
] NexusDriver: 
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
<interface>
    <nve>nve1</nve>
    <__XML__MODE_if-nve>
        <member>no member vni 222224</member>
    </__XML__MODE_if-nve>
</interface>

</__XML__MODE__exec_configure>
        </configure>
      </config>
 from (pid=31927) delete_nve_member /opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:380
2015-03-05 10:33:03.610 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [req-27f75e6b-c7eb-4e8e-8979-804c2dc34d4d admin 05e1512cadd84617a8cb4bc1f3e64c80
] NexusDriver config: 
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
<interface>
    <nve>nve1</nve>
    <__XML__MODE_if-nve>
        <member>no member vni 222224</member>
    </__XML__MODE_if-nve>
</interface>

</__XML__MODE__exec_configure>
        </configure>
      </config>
 from (pid=31927) _edit_config /opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:99
2015-03-05 10:33:03.612 INFO neutron.wsgi [req-5f6c2b8e-f759-4e88-ba7f-b6ae361b303e admin 05e1512cadd84617a8cb4bc1f3e64c80] 172.29.172.163 - - [05/Mar/
2015 10:33:03] "DELETE /v2.0/ports/ab786270-ed85-4439-afe4-672c05555b82.json HTTP/1.1" 204 149 2.072496
2015-03-05 10:33:03.840 ERROR neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [req-27f75e6b-c7eb-4e8e-8979-804c2dc34d4d admin 05e1512cadd84617a8cb4bc1f3e64c80
] Nexus Driver cisco_nexus failed in (delete_nve_member||disable_vxlan_feature)
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus Traceback (most recent call last):
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py",
 line 364, in _delete_nve_member
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     const.NVE_INT_NUM, vni)
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.
py", line 381, in delete_nve_member
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     self._edit_config(nexus_host, config=confstr)
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus   File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.
py", line 116, in _edit_config
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     exc=e)
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus NexusConfigFailed: Failed to configure Nexus switch: 172.20.231.7 XML: 
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus       <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus         <configure>
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus           <__XML__MODE__exec_configure>
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <interface>
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     <nve>nve1</nve>
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     <__XML__MODE_if-nve>
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus         <member>no member vni 222224</member>
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     </__XML__MODE_if-nve>
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </interface>
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus 
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus           </__XML__MODE__exec_configure>
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus         </configure>
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus       </config>
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus . Reason: ERROR: VNI delete failed: Could not find vni node for 222224
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus .
2015-03-05 10:33:03.840 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus

Revision history for this message

Danny Choi (dannchoi) wrote on 2015-04-21:

Download full text (17.8 KiB)

This traceback still happens in Kilo.

Below is the log when deleting 2VMs with the same tenant network from a compute node:

2015-04-21 08:21:26.045 DEBUG networking_cisco.plugins.ml2.drivers.cisco.nexus.nexus_db_v2 [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] get_nve_vni_switch_bindings() called from (pid=20662) get_nve_vni_switch_bindings /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_db_v2.py:195
2015-04-21 08:21:26.047 DEBUG networking_cisco.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] NexusDriver: from (pid=20662) delete_nve_member /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:392
2015-04-21 08:21:26.047 DEBUG networking_cisco.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] NexusDriver edit config:
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
<interface>
    <nve>nve1</nve>
    <__XML__MODE_if-nve>
        <member>no member vni 12345</member>
    </__XML__MODE_if-nve>
</interface>

          </__XML__MODE__exec_configure>
        </configure>
      </config>
from (pid=20662) _edit_config /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:107
2015-04-21 08:21:26.079 DEBUG oslo_messaging._drivers.amqp [-] unpacked context: {u'read_only': False, u'domain': None, u'project_name': None, u'user_id': None, u'show_deleted': False, u'roles': [u'admin'], u'user_identity': u'- - - - -', u'project_domain': None, u'tenant_name': None, u'auth_token': None, u'resource_uuid': None, u'project_id': None, u'tenant_id': None, u'is_admin': True, u'user': None, u'request_id': u'req-77049958-52c2-4047-9f45-5ddd743020ca', u'tenant': None, u'user_domain': None, u'timestamp': u'2015-04-20 20:37:13.854969', u'read_deleted': u'no', u'user_name': None} from (pid=20662) unpack_context /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:203
2015-04-21 08:21:26.080 DEBUG neutron.plugins.ml2.rpc [req-77049958-52c2-4047-9f45-5ddd743020ca None None] Device 7505f47b-6d06-4589-96bf-b5eaff2ad5be no longer exists at agent ovs-agent-qa6 from (pid=20662) update_device_down /opt/stack/neutron/neutron/plugins/ml2/rpc.py:144
2015-04-21 08:21:26.083 DEBUG neutron.plugins.ml2.plugin [req-77049958-52c2-4047-9f45-5ddd743020ca None None] No Port match for: 7505f47b-6d06-4589-96bf-b5eaff2ad5be from (pid=20662) port_bound_to_host /opt/stack/neutron/neutron/plugins/ml2/plugin.py:1435
2015-04-21 08:21:26.083 DEBUG neutron.plugins.ml2.rpc [req-77049958-52c2-4047-9f45-5ddd743020ca None None] Device 7505f47b-6d06-4589-96bf-b5eaff2ad5be not bound to the agent host qa6 from (pid=20662) update_device_down /opt/stack/neutron/neutron/plugins/ml2/rpc.py:152
2015-04-21 08:21:26.084 DEBUG oslo_messaging._drivers.amqp [req-77049958-52c2-4047-9f45-5ddd743020ca None None] UNIQUE_ID is a5da0275b34446febd11acce6c377e4e. from ...

This traceback still happens in Kilo.

Below is the log when deleting 2VMs with the same tenant network from a compute node:

2015-04-21 08:21:26.045 DEBUG networking_cisco.plugins.ml2.drivers.cisco.nexus.nexus_db_v2 [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] get_nve_vni_switch_bindings() called from (pid=20662) get_nve_vni_switch_bindings /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_db_v2.py:195
2015-04-21 08:21:26.047 DEBUG networking_cisco.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] NexusDriver:  from (pid=20662) delete_nve_member /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:392
2015-04-21 08:21:26.047 DEBUG networking_cisco.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] NexusDriver edit config: 
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
<interface>
    <nve>nve1</nve>
    <__XML__MODE_if-nve>
        <member>no member vni 12345</member>
    </__XML__MODE_if-nve>
</interface>

</__XML__MODE__exec_configure>
        </configure>
      </config>
 from (pid=20662) _edit_config /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:107
2015-04-21 08:21:26.079 DEBUG oslo_messaging._drivers.amqp [-] unpacked context: {u'read_only': False, u'domain': None, u'project_name': None, u'user_id': None, u'show_deleted': False, u'roles': [u'admin'], u'user_identity': u'- - - - -', u'project_domain': None, u'tenant_name': None, u'auth_token': None, u'resource_uuid': None, u'project_id': None, u'tenant_id': None, u'is_admin': True, u'user': None, u'request_id': u'req-77049958-52c2-4047-9f45-5ddd743020ca', u'tenant': None, u'user_domain': None, u'timestamp': u'2015-04-20 20:37:13.854969', u'read_deleted': u'no', u'user_name': None} from (pid=20662) unpack_context /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:203
2015-04-21 08:21:26.080 DEBUG neutron.plugins.ml2.rpc [req-77049958-52c2-4047-9f45-5ddd743020ca None None] Device 7505f47b-6d06-4589-96bf-b5eaff2ad5be no longer exists at agent ovs-agent-qa6 from (pid=20662) update_device_down /opt/stack/neutron/neutron/plugins/ml2/rpc.py:144
2015-04-21 08:21:26.083 DEBUG neutron.plugins.ml2.plugin [req-77049958-52c2-4047-9f45-5ddd743020ca None None] No Port match for: 7505f47b-6d06-4589-96bf-b5eaff2ad5be from (pid=20662) port_bound_to_host /opt/stack/neutron/neutron/plugins/ml2/plugin.py:1435
2015-04-21 08:21:26.083 DEBUG neutron.plugins.ml2.rpc [req-77049958-52c2-4047-9f45-5ddd743020ca None None] Device 7505f47b-6d06-4589-96bf-b5eaff2ad5be not bound to the agent host qa6 from (pid=20662) update_device_down /opt/stack/neutron/neutron/plugins/ml2/rpc.py:152
2015-04-21 08:21:26.084 DEBUG oslo_messaging._drivers.amqp [req-77049958-52c2-4047-9f45-5ddd743020ca None None] UNIQUE_ID is a5da0275b34446febd11acce6c377e4e. from (pid=20662) _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-21 08:21:26.085 DEBUG oslo_messaging._drivers.amqp [req-77049958-52c2-4047-9f45-5ddd743020ca None None] UNIQUE_ID is d13dbda039e043c9bfbb6495e19764a6. from (pid=20662) _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-21 08:21:26.092 DEBUG oslo_messaging._drivers.amqp [-] unpacked context: {u'read_only': False, u'domain': None, u'project_name': None, u'user_id': None, u'show_deleted': False, u'roles': [u'admin'], u'user_identity': u'- - - - -', u'project_domain': None, u'tenant_name': None, u'auth_token': None, u'resource_uuid': None, u'project_id': None, u'tenant_id': None, u'is_admin': True, u'user': None, u'request_id': u'req-77049958-52c2-4047-9f45-5ddd743020ca', u'tenant': None, u'user_domain': None, u'timestamp': u'2015-04-20 20:37:13.854969', u'read_deleted': u'no', u'user_name': None} from (pid=20662) unpack_context /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:203
2015-04-21 08:21:26.094 DEBUG neutron.plugins.ml2.rpc [req-77049958-52c2-4047-9f45-5ddd743020ca None None] Device 76e21dd5-d0f6-4873-9d1c-ec88ed3fbe99 no longer exists at agent ovs-agent-qa6 from (pid=20662) update_device_down /opt/stack/neutron/neutron/plugins/ml2/rpc.py:144
2015-04-21 08:21:26.100 DEBUG neutron.plugins.ml2.plugin [req-77049958-52c2-4047-9f45-5ddd743020ca None None] No Port match for: 76e21dd5-d0f6-4873-9d1c-ec88ed3fbe99 from (pid=20662) port_bound_to_host /opt/stack/neutron/neutron/plugins/ml2/plugin.py:1435
2015-04-21 08:21:26.100 DEBUG neutron.plugins.ml2.rpc [req-77049958-52c2-4047-9f45-5ddd743020ca None None] Device 76e21dd5-d0f6-4873-9d1c-ec88ed3fbe99 not bound to the agent host qa6 from (pid=20662) update_device_down /opt/stack/neutron/neutron/plugins/ml2/rpc.py:152
2015-04-21 08:21:26.100 DEBUG oslo_messaging._drivers.amqp [req-77049958-52c2-4047-9f45-5ddd743020ca None None] UNIQUE_ID is 18f51ad95cef46beb4645ca7ac191c23. from (pid=20662) _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-21 08:21:26.101 DEBUG oslo_messaging._drivers.amqp [req-77049958-52c2-4047-9f45-5ddd743020ca None None] UNIQUE_ID is 058c908fb0b74d8fb6c578e06c7b4599. from (pid=20662) _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-21 08:21:26.162 DEBUG networking_cisco.plugins.ml2.drivers.cisco.nexus.nexus_db_v2 [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] get_port_vlan_switch_binding() called from (pid=20662) get_port_vlan_switch_binding /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_db_v2.py:109
2015-04-21 08:21:26.165 DEBUG networking_cisco.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] NexusDriver edit config: 
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
          <interface>
            <ethernet>
              <interface>1/35</interface>
              <__XML__MODE_if-ethernet-switch>
                <switchport></switchport>
                <switchport>
                  <trunk>
                    <allowed>
                      <vlan>
                        <remove>
                          <vlan>300</vlan>
                        </remove>
                      </vlan>
                    </allowed>
                  </trunk>
                </switchport>
              </__XML__MODE_if-ethernet-switch>
            </ethernet>
          </interface>

</__XML__MODE__exec_configure>
        </configure>
      </config>
 from (pid=20662) _edit_config /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:107
2015-04-21 08:21:26.330 DEBUG networking_cisco.plugins.ml2.drivers.cisco.nexus.nexus_db_v2 [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] get_nexusvlan_binding() called from (pid=20662) get_nexusvlan_binding /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_db_v2.py:42
2015-04-21 08:21:26.334 DEBUG networking_cisco.plugins.ml2.drivers.cisco.nexus.nexus_network_driver [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] NexusDriver edit config: 
      <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
        <configure>
          <__XML__MODE__exec_configure>
          <no>
          <vlan>
            <vlan-id-create-delete>
              <__XML__PARAM_value>300</__XML__PARAM_value>
            </vlan-id-create-delete>
          </vlan>
          </no>

</__XML__MODE__exec_configure>
        </configure>
      </config>
 from (pid=20662) _edit_config /usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py:107
2015-04-21 08:21:26.469 DEBUG neutron.policy [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] Enforcing rules: ['get_port:binding:host_id'] from (pid=20662) log_rule_list /opt/stack/neutron/neutron/policy.py:363
2015-04-21 08:21:26.469 DEBUG neutron.policy [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] Enforcing rules: ['get_port:binding:profile'] from (pid=20662) log_rule_list /opt/stack/neutron/neutron/policy.py:363
2015-04-21 08:21:26.470 DEBUG neutron.policy [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] Enforcing rules: ['get_port:binding:vif_details'] from (pid=20662) log_rule_list /opt/stack/neutron/neutron/policy.py:363
2015-04-21 08:21:26.470 DEBUG neutron.policy [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] Enforcing rules: ['get_port:binding:vif_type'] from (pid=20662) log_rule_list /opt/stack/neutron/neutron/policy.py:363
2015-04-21 08:21:26.489 DEBUG oslo_messaging._drivers.amqp [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] UNIQUE_ID is 22c3522fb8c24fd19a3beae812a590ac. from (pid=20662) _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:258
2015-04-21 08:21:26.492 INFO neutron.wsgi [req-d5c56275-d893-403d-9aa6-a6f5f08dbda0 demo 142fa7f81fba4f729d2004a88b3ee364] 172.29.172.163 - - [21/Apr/2015 08:21:26] "DELETE /v2.0/ports/7505f47b-6d06-4589-96bf-b5eaff2ad5be.json HTTP/1.1" 204 149 0.701544
2015-04-21 08:21:26.514 INFO neutron.wsgi [-] (20662) accepted ('172.29.172.161', 47110)
2015-04-21 08:21:26.517 DEBUG keystoneclient.session [-] REQ: curl -g -i --cacert "/opt/stack/data/ca-bundle.pem" -X GET http://172.29.172.161:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}97c5927de4fc9fd7abbbe3f7b47f027bc2b62538" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}5321aadcfb08f2a5f4c7088dd0157a8a0aef9215" from (pid=20662) _http_log_request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:190
2015-04-21 08:21:26.607 DEBUG keystoneclient.session [-] RESP: [200] content-length: 5483 x-subject-token: {SHA1}97c5927de4fc9fd7abbbe3f7b47f027bc2b62538 vary: X-Auth-Token server: Apache/2.4.7 (Ubuntu) date: Tue, 21 Apr 2015 12:21:26 GMT content-type: application/json x-openstack-request-id: req-a3c95c19-522d-4b21-a081-ca51cb1a5e0d 
RESP BODY: {"token": {"methods": ["password", "token"], "roles": [{"id": "7bf9ebbef2154f25a15e471842e50717", "name": "Member"}, {"id": "2329d4e626614e12af525a44febf619f", "name": "anotherrole"}], "expires_at": "2015-04-21T13:21:26.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "142fa7f81fba4f729d2004a88b3ee364", "name": "demo"}, "catalog": "<removed>", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "e3a2b5981e9e4f9eab912d5d718ce67a", "name": "demo"}, "audit_ids": ["Azr9C0erRwKpyoDauc9GCg"], "issued_at": "2015-04-21T12:21:26.227256"}}
 from (pid=20662) _http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:218
2015-04-21 08:21:26.630 INFO neutron.wsgi [req-27e08a8f-beee-4a05-bbae-0c0f3dd66754 demo 142fa7f81fba4f729d2004a88b3ee364] 172.29.172.161 - - [21/Apr/2015 08:21:26] "GET /v2.0/ports.json?device_id=4e0b3645-35d5-4871-9cb3-b48c0f2b0c09&device_id=3dd637e0-88c2-40f7-90b4-660738baca8d HTTP/1.1" 200 202 0.115564
2015-04-21 08:21:26.680 ERROR networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [req-6d8dead7-201c-4636-a57a-b0e6ff053cd0 demo 142fa7f81fba4f729d2004a88b3ee364] Nexus Driver cisco_nexus failed in (delete_nve_member||disable_vxlan_feature)
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus Traceback (most recent call last):
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus   File "/usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py", line 364, in _delete_nve_member
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     const.NVE_INT_NUM, vni)
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus   File "/usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 393, in delete_nve_member
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     self._edit_config(nexus_host, config=confstr)
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus   File "/usr/local/lib/python2.7/dist-packages/networking_cisco/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 124, in _edit_config
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     exc=e)
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus NexusConfigFailed: Failed to configure Nexus switch: 172.20.231.7 XML: 
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus       <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus         <configure>
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus           <__XML__MODE__exec_configure>
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <interface>
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     <nve>nve1</nve>
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     <__XML__MODE_if-nve>
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus         <member>no member vni 12345</member>
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus     </__XML__MODE_if-nve>
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </interface>
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus 
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus           </__XML__MODE__exec_configure>
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus         </configure>
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus       </config>
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus . Reason: ERROR: VNI delete failed: Could not find vni node for 12345
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus .
2015-04-21 08:21:26.680 TRACE networking_cisco.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus 
2015-04-21 08:21:26.693 DEBUG oslo_concurrency.lockutils [req-6d8dead7-201c-4636-a57a-b0e6ff053cd0 demo 142fa7f81fba4f729d2004a88b3ee364] Lock "cisco-nexus-portlock" released by "delete_port_postcommit" :: held 0.241s from (pid=20662) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

screen-q-svc.log Edit

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.