Nexus plugins fails to configure the network node connection when the network node is rebooted before launching the VM

[Danny Choi]

In the ml2 .ini file, I have the network node entry defined separately:

[ml2_mech_cisco_nexus:10.86.7.59]
bxb-ds-46=port-channel:1 <<<<< network node
bxb-ds-47=port-channel:2
bxb-ds-48=port-channel:3
ssh_port=22
username=admin
password=cisco123

I have one tenant network defined.
Without any VM, the dhcp namespace is not created (expected behavior), and nothing in DB:

[root@bxb-ds-46 ~]# mysql -e "use neutron; select * from cisco_ml2_nexusport_bindings;"
[root@bxb-ds-46 ~]# ip netns
qrouter-c4ec7fa5-12b8-4831-a4f5-12291b30dfc1
[root@bxb-ds-46 ~]# neutron net-list
+--------------------------------------+----------------+-----------------------------------------------------+
| id | name | subnets |
+--------------------------------------+----------------+-----------------------------------------------------+
| cac46af7-9441-4589-9818-9126d9736877 | test_network_1 | 9c1d3bd2-bfdb-48fb-aaff-a1c399c0a000 192.168.1.0/24 |
| 590ae6a2-600b-475d-a88a-d91526ce3471 | ext-net | ebc5027f-7ac9-497d-b60f-c23d62b206bd 10.86.7.0/26 |
+--------------------------------------+----------------+-----------------------------------------------------+
[root@bxb-ds-46 ~]# neutron subnet-list
+--------------------------------------+---------------+----------------+--------------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+---------------+----------------+--------------------------------------------------+
| 9c1d3bd2-bfdb-48fb-aaff-a1c399c0a000 | test_subnet_1 | 192.168.1.0/24 | {"start": "192.168.1.2", "end": "192.168.1.254"} |
| ebc5027f-7ac9-497d-b60f-c23d62b206bd | | 10.86.7.0/26 | {"start": "10.86.7.20", "end": "10.86.7.22"} |
+--------------------------------------+---------------+----------------+--------------------------------------------------+

Now I reboot the network node (which is also the controller node).
Upon boots up, the dhcp namespace is created while the DB is still empty:

[root@bxb-ds-46 ~]# reboot
Connection to 10.86.7.46 closed by remote host.
Connection to 10.86.7.46 closed.
DANNCHOI-M-G07T:~ dannychoi$ ssh root@10.86.7.46
root@10.86.7.46's password:
Last login: Thu Jun 4 10:05:24 2015 from bxb-ads-504.cisco.com
[root@bxb-ds-46 ~]# ip netns
qrouter-c4ec7fa5-12b8-4831-a4f5-12291b30dfc1
qdhcp-cac46af7-9441-4589-9818-9126d9736877
[root@bxb-ds-46 ~]# mysql -e "use neutron; select * from cisco_ml2_nexusport_bindings;"
[root@bxb-ds-46 ~]#

Issue is that if I launch a VM now, it does not configure the network node connection at the Nexus switch (port-cahnnel:1):

[root@bxb-ds-46 ~]# nova list
+--------------------------------------+----------+--------+------------+-------------+----------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+----------+--------+------------+-------------+----------------------------+
| c5cc9ebe-a35c-40da-98a4-a4ef7a98263f | ds47-vm1 | ACTIVE | - | Running | test_network_1=192.168.1.3 |
+--------------------------------------+----------+--------+------------+-------------+----------------------------+
[root@bxb-ds-46 ~]# mysql -e "use neutron; select * from cisco_ml2_nexusport_bindings;"
+------------+----------------+---------+-------------+--------------------------------------+------+------------------+
| binding_id | port_id | vlan_id | switch_ip | instance_id | vni | is_provider_vlan |
+------------+----------------+---------+-------------+--------------------------------------+------+------------------+
| 1 | port-channel:2 | 462 | 10.86.7.59 | c5cc9ebe-a35c-40da-98a4-a4ef7a98263f | 0 | 0 |
| 2 | port-channel:2 | 462 | 10.86.1.128 | c5cc9ebe-a35c-40da-98a4-a4ef7a98263f | 0 | 0 |
+------------+----------------+---------+-------------+--------------------------------------+------+------------------+

[Timothy Swanson (tiswanso)]

Symptom:
   If the network node is rebooted,

       the following would be affected:
          1) creation of new VMs on tenant networks that existed prior to network
              node reboot would not get their VLAN config on the switch
     - Since the dhcp namespaces for these networks are already created
     after the reboot.

       the following would be unaffected:
          1) any existing VMs
              - since the tenant network's VLAN has already been configured on the switch
          2) any new VMs created on tenant networks already having VMs
              - since the tenant network's VLAN has already been configured on the switch
          3) any new tenant networks and their VMs created after the network node is back

Cause:
 Upon boots up, the dhcp namespace is created while the DB is still empty
   - VM triggered namespace creation is path where Nexus plugin configures the
      VLAN

[Timothy Swanson (tiswanso)]

******Workaround**********

To limit the number of VLANs configured at that trunk port where the neutron server is hanging off,
the other way around is to include this port as a list of ports associate with each compute node.

E.g. Network node is port 1, compute-1 is port 2, compute-2 is port 3
[ml2_mech_cisco_nexus:10.86.7.59]
compute-1=1/1,1/2
compute-2=1/1,1/3
ssh_port=22
username=admin
password=cisco123

This way the nexus plugin still manages the trunk configuration of the switch port connecting the network node and only allows the VLANs associated with tenant networks.

[Danny Choi (dannchoi)]

While verifying this workaround:

[ml2_mech_cisco_nexus:10.86.7.59]
compute-1=1/1,1/2
compute-2=1/1,1/3
ssh_port=22
username=admin
password=cisco123

I came across another issue.

When the DHCP namespace is automatically created after the network node reboot, there is a port binding failure for the corresponding DHCP neutron port:

[root@bxb-ds-46 ~]# ip netns
qdhcp-d1e57ac4-b775-4dd1-ab4d-c901ebb04c98
qrouter-3b3f07ff-f530-4cfb-8a4b-f85ea0d4441c
[root@bxb-ds-46 ~]# neutron port-list
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------------+
| 036acf84-a5c2-4789-a742-bc049d0d2dff | | fa:16:3e:72:c2:8e | {"subnet_id": "768ea98d-3b7e-43a5-b18a-d3a1a284c193", "ip_address": "10.86.7.239"} |
| 52a2db1a-1929-4c2c-8720-c08b0706b4cb | | fa:16:3e:d1:43:a6 | {"subnet_id": "e7ec9460-b1aa-4801-a128-345955a38467", "ip_address": "192.168.1.2"} |
| 7a4610d4-9a63-4eee-88ea-8086cb5702fc | | fa:16:3e:fb:98:0d | {"subnet_id": "e7ec9460-b1aa-4801-a128-345955a38467", "ip_address": "192.168.1.1"} |
+--------------------------------------+------+-------------------+——————————————————————————————————————————+
[root@bxb-ds-46 ~]# neutron port-show 52a2db1a-1929-4c2c-8720-c08b0706b4cb
+-----------------------+------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+------------------------------------------------------------------------------------+
| admin_state_up | True |
| allowed_address_pairs | |
| binding:host_id | bxb-ds-46 |
| binding:profile | {} |
| binding:vif_details | {} |
| binding:vif_type | binding_failed | <<<<<ERROR
| binding:vnic_type | normal |
| device_id | dhcpc63a1974-ca08-50bd-9dfd-34e0a562acaf-d1e57ac4-b775-4dd1-ab4d-c901ebb04c98 |
| device_owner | network:dhcp |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "e7ec9460-b1aa-4801-a128-345955a38467", "ip_address": "192.168.1.2"} |
| id | ...

[Danny Choi (dannchoi)]

Remove the cisco_nexus driver in ml2conf.ini file and saw the same issues after the Network node reboot.

1. DHCP namespace is created automatically.
2. DHCP port binding failure.

/etc/neutron/plugins/ml2/ml2_conf.ini:
mechanism_drivers =openvswitch /* remove cisco_nexus */

[Danny Choi (dannchoi)]

Retest on RHEL reference test bed without Cisco neutron:

[root@hh24-11 ~]# rpm -qi python-neutron
Name : python-neutron
Version : 2014.2.3
Release : 2.el7ost
Architecture: noarch
Install Date: Thu 16 Jul 2015 11:17:46 AM PDT
Group : Applications/System
Size : 11211657
License : ASL 2.0
Signature : RSA/SHA256, Sun 26 Apr 2015 08:03:39 AM PDT, Key ID 199e2f91fd431d51
Source RPM : openstack-neutron-2014.2.3-2.el7ost.src.rpm
Build Date : Mon 13 Apr 2015 06:23:08 AM PDT
Build Host : x86-021.build.eng.bos.redhat.com
Relocations : (not relocatable)
Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor : Red Hat, Inc.
URL : http://launchpad.net/neutron/
Summary : Neutron Python libraries

Test result:
1. DHCP namespace is created automatically after a reboot or neutron restart.
2. DHCP port binding is successful.

Before reboot:
==============
[root@hh24-11 ~]# neutron net-list
+--------------------------------------+---------+----------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+----------------------------------------------------+
| 0a7cafba-100d-4761-9799-96c32bd54cd5 | ext-net | 66a2a867-af61-4017-bd94-54f04d973073 2.2.2.0/24 |
| 1e0b9add-3df6-483c-840a-533b4323c2bc | private | c2e11c64-2269-486f-b1ce-15c358db129a 10.10.10.0/24 |
+--------------------------------------+---------+----------------------------------------------------+
[root@hh24-11 ~]# neutron subnet-list
+--------------------------------------+----------------+---------------+------------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+----------------+---------------+------------------------------------------------+
| 66a2a867-af61-4017-bd94-54f04d973073 | | 2.2.2.0/24 | {"start": "2.2.2.2", "end": "2.2.2.200"} |
| c2e11c64-2269-486f-b1ce-15c358db129a | private-subnet | 10.10.10.0/24 | {"start": "10.10.10.2", "end": "10.10.10.254"} |
+--------------------------------------+----------------+---------------+------------------------------------------------+
[root@hh24-11 ~]# neutron port-list
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+------+-------------------+-----------------------------------------------------------------------------------+
| 79acd252-7f56-461c-be2d-3b38ab0d878c | | fa:16:3e:5a:98:51 | {"subnet_id": "66a2a867-af61-4017-bd94-54f04d973073", "ip_address": "2.2.2.2"} |
| 98e09da6-104e-42e8-8802-39aaf02e3e3d | | fa:16:3e:24:2c:01 | {"subnet_id": "c2e11c64-2269-486f-b1ce-15c358db129a", "ip_address": "10.10.10.1"} |
+--------------------------------------+------...