Inconsistent behavior regrading VM creation/deletion while the Nexus switch is rebooting.
1. Create a new VM using a distinct network:
- VM is created successfully; in Active/Running state, and Nova reports a IP address assigned.
- entry created in mysql database.
- When the Nexus switch boots up, the replay code configures the VLAN associated with that network.
2. Delete the last VM using that network:
- VM fails to be deleted; in ERROR/NOSTATE.
- entry removed in mysql database.
- traceback is logged in screen-q-svc.log
- When the Nexus switch boots up, the replay code does not configured the VLAN associated with that network.
- Have to issue the delete command to cleanly remove the VM.
3. Delete the last VM using that network and create a new VM using the same network:
- Same behavior with the VM deletion as above.
- Fail to create the new VM; in ERROR/NOSTATE, Nova reports no IP address assigned.
- no entry in mysql database.
- error is logged in screen-n-cone.log.
- traceback is logged in screen-q-svc.log.
- When the Nexus switch boots up, the replay code does not configure the VLAN associated with that network.
*******************
1. Traceback during VM creation:
2015-03-16 12:44:15.901 ERROR neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] Lost connection to switch ip 172.20.231.7
2015-03-16 12:44:15.901 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] check_connections() switch 172.20.231.5 state 1 from (pid=29820) check_connect
ions /opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py:121
2015-03-16 12:44:15.903 ERROR neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [req-b3c88934-a482-4af4-80eb-905aec34f94e None None] Nexus Driver cisco_nexus failed in create_nve_member
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus Traceback (most recent call last):
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py", line 337, in _configure_nve_member
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus vni, mcast_group)
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 376, in create_nve_member
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus self._edit_config(nexus_host, config=confstr)
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 117, in _edit_config
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus exc=e)
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus NexusConfigFailed: Failed to configure Nexus switch: 172.20.231.7 XML:
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <configure>
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <__XML__MODE__exec_configure>
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <interface>
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <nve>nve1</nve>
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <__XML__MODE_if-nve>
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <member>member vni 12361 mcast-group 225.1.1.2</member>
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </__XML__MODE_if-nve>
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </interface>
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </__XML__MODE__exec_configure>
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </configure>
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </config>
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus . Reason: .
2015-03-16 12:44:15.903 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus
2. Traceback during VM deletion:
2015-03-16 12:50:24.897 ERROR neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] Lost connection to switch ip 172.20.231.7
2015-03-16 12:50:24.897 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] check_connections() switch 172.20.231.5 state 1 from (pid=29820) check_connect
ions /opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py:121
2015-03-16 12:50:24.919 ERROR neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [req-4804eac6-bc6d-41c8-81e4-603ce8112966 demo 7f7caff2297f44b7a822eb1bb133a8bf] Nexus Driver cisco_nexus failed in (delete_nve_member||disable_vxlan_feature)
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus Traceback (most recent call last):
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py", line 366, in _delete_nve_member
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus const.NVE_INT_NUM, vni)
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 383, in delete_nve_member
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus self._edit_config(nexus_host, config=confstr)
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 117, in _edit_config
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus exc=e)
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus NexusConfigFailed: Failed to configure Nexus switch: 172.20.231.7 XML:
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <configure>
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <__XML__MODE__exec_configure>
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <interface>
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <nve>nve1</nve>
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <__XML__MODE_if-nve>
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <member>no member vni 12361</member>
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </__XML__MODE_if-nve>
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </interface>
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </__XML__MODE__exec_configure>
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </configure>
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </config>
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus . Reason: .
2015-03-16 12:50:24.919 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus
3. ERROR in screen-n-cond.log and Traceback in screen-q-svc.log during VM delete and create:
screen-n-cone.log:
================
2015-03-16 13:18:31.751 ERROR nova.scheduler.utils [req-f4170032-33c2-413b-ba0e-652997ab3521 demo demo] [instance: f18d8665-a63b-4009-88cc-0c3abe4ffcb0] Error from last host
: qa6 (node qa6): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2033, in _do_build_and_run_instance\n filter_properties)\n', u' File "/opt/stack/nova/nova/com
pute/manager.py", line 2164, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance f18d8665-a63b-4009-88cc-0c3abe4ffcb0 was re-scheduled:
Connection to neutron failed: HTTPConnectionPool(host='172.29.172.161', port=9696): Read timed out. (read timeout=30)\n"]
2015-03-16 13:18:31.753 INFO oslo.messaging._drivers.impl_rabbit [req-f4170032-33c2-413b-ba0e-652997ab3521 demo demo] Connecting to AMQP server on 172.29.172.161:5672
2015-03-16 13:18:31.761 INFO oslo.messaging._drivers.impl_rabbit [req-f4170032-33c2-413b-ba0e-652997ab3521 demo demo] Connected to AMQP server on 172.29.172.161:5672
2015-03-16 13:18:31.793 WARNING nova.scheduler.driver [req-f4170032-33c2-413b-ba0e-652997ab3521 demo demo] [instance: f18d8665-a63b-4009-88cc-0c3abe4ffcb0] NoValidHost excep
tion with message: 'No valid host was found.'
2015-03-16 13:18:31.793 WARNING nova.scheduler.driver [req-f4170032-33c2-413b-ba0e-652997ab3521 demo demo] [instance: f18d8665-a63b-4009-88cc-0c3abe4ffcb0] Setting instance
to ERROR state.
screen-q-svc.log:
===============
2015-03-16 13:18:42.436 ERROR neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] Lost connection to switch ip 172.20.231.7
2015-03-16 13:18:42.437 DEBUG neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [-] check_connections() switch 172.20.231.5 state 1 from (pid=29820) check_connections /opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py:121
2015-03-16 13:18:42.459 ERROR neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus [req-8ed464eb-7aad-44eb-8c0b-4bf72d17f581 demo 7f7caff2297f44b7a822eb1bb133a8bf] Nexus Driver cisco_nexus failed in (delete_nve_member||disable_vxlan_feature)
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus Traceback (most recent call last):
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/mech_cisco_nexus.py", line 366, in _delete_nve_member
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus const.NVE_INT_NUM, vni)
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 383, in delete_nve_member
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus self._edit_config(nexus_host, config=confstr)
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus File "/opt/stack/neutron/neutron/plugins/ml2/drivers/cisco/nexus/nexus_network_driver.py", line 117, in _edit_config
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus exc=e)
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus NexusConfigFailed: Failed to configure Nexus switch: 172.20.231.7 XML:
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <configure>
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <__XML__MODE__exec_configure>
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <interface>
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <nve>nve1</nve>
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <__XML__MODE_if-nve>
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus <member>no member vni 12361</member>
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </__XML__MODE_if-nve>
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </interface>
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </__XML__MODE__exec_configure>
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </configure>
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus </config>
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus . Reason: .
2015-03-16 13:18:42.459 TRACE neutron.plugins.ml2.drivers.cisco.nexus.mech_cisco_nexus
