[DVS] Error occurs when creating neutron subnet with vmware nsx plugin

[youngnoh kim]

Summary : [DVS] Error occurs when creating neutron subnet with vmware nsx plugin

Decription
==========
Error occurs when creating neutron subnet with vmware nsx plugin.
Then vm does not get ip from neutron dhcp agent.

Environment
===========
Openstack Controller & Compute 1ea (CentOS Linux release 7.3.1611 / 3.10.0-514.26.2.el7.x86_64)
VMware ESXi 6.5 2ea

OpenStack Version (newton release)
 - openstack-neutron-9.4.1-1.el7.noarch
 - openvswitch-2.6.1-10.1.git20161206.el7.x86_64
 - python-networking-vmware-nsx-9.1.0-1.el7.noarch

Configuration summary
$ cat /etc/neutron/neutron.conf
[DEFAULT]
...
core_plugin = vmware_nsx.plugin.NsxDvsPlugin
...

$ cat /etc/neutron/plugins/vmware/nsx.ini
[dvs]
host_ip = <vcenter ip>
host_username = <email address hidden>
host_password = <vcenter password>
dvs_name = DSwitch
insecure = true

$ cat /etc/neutron/dhcp_agent.ini
[DEFAULT]
...
ovs_integration_bridge = br-dvs
enable_isolated_metadata = true
enable_metadata_network = true
...

$ cat /etc/neutron/plugins/ml2/openvswitch_agent.ini
[ovs]
...
integration_bridge = br-dvs
bridge_mappings =
...

$ cat /etc/nova/nova.conf
[DEFAULT]
...
compute_driver = vmwareapi.VMwareVCDriver
...

[vmware]
host_ip = <vcenter ip>
host_username = <email address hidden>
host_password = <vcenter password>
cluster_name = TestCluster
datastore_regex = "VMFS.*"
insecure = true

Reproduce
=========
$ neutron agent-list
+--------------------------------------+--------------------+--------------+-------+----------------+---------------------------+
| id | agent_type | host | alive | admin_state_up | binary |
+--------------------------------------+--------------------+--------------+-------+----------------+---------------------------+
| 20a968ac-7d05-4a06-af92-8ade7a330be9 | Metadata agent | controller01 | :-) | True | neutron-metadata-agent |
| b2d78dad-bd1a-4c98-a2f1-507b7157f71c | Open vSwitch agent | controller01 | :-) | True | neutron-openvswitch-agent |
| f4318806-3a6c-40f8-8a99-446e7c3c08f1 | DHCP agent | controller01 | :-) | True | neutron-dhcp-agent |
+--------------------------------------+--------------------+--------------+-------+----------------+---------------------------+

$ neutron net-create --tenant-id 17e3939cff16474f9da664479bf5248d \
 --provider:network_type vlan \
 --provider:phical_network dvs \
 --provider:segmentation_id 175 \
 test-network75
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| created_at | 2018-01-26T03:23:38Z |
| description | |
| id | 168469fc-bf62-49d1-a823-2802be620876 |
| name | test-network75 |
| port_security_enabled | True |
| project_id | 17e3939cff16474f9da664479bf5248d |
| provider:network_type | vlan |
| provider:physical_network | dvs |
| provider:segmentation_id | 175 |
| revision_number | 2 |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | |
| tags | |
| tenant_id | 17e3939cff16474f9da664479bf5248d |
| updated_at | 2018-01-26T03:23:38Z |
+---------------------------+--------------------------------------+

$ neutron subnet-create --tenant-id 17e3939cff16474f9da664479bf5248d \
 --name test-subnet75 \
 --gateway 10.168.75.1 \
 --allocation-pool start=10.168.75.11,end=10.168.75.254 \
 --enable-dhcp \
 test-network75 10.168.75.0/24
Created a new subnet:
+-------------------+---------------------------------------------------+
| Field | Value |
+-------------------+---------------------------------------------------+
| allocation_pools | {"start": "10.168.75.11", "end": "10.168.75.254"} |
| cidr | 10.168.75.0/24 |
| created_at | 2018-01-26T03:28:46Z |
| description | |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.168.75.1 |
| host_routes | |
| id | caa4a01b-2d91-4f5e-bf32-bc540c2a0708 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | test-subnet75 |
| network_id | 168469fc-bf62-49d1-a823-2802be620876 |
| project_id | 17e3939cff16474f9da664479bf5248d |
| revision_number | 2 |
| subnetpool_id | |
| tenant_id | 17e3939cff16474f9da664479bf5248d |
| updated_at | 2018-01-26T03:28:46Z |
+-------------------+---------------------------------------------------+

$ ovs-vsctl show
ba18cda3-7b92-4dc0-b0cc-5ecfcb6e99c5
    Manager "ptcp:6640:127.0.0.1"
    Bridge br-dvs
        fail_mode: secure
        Port "bond0"
            Interface "bond0"
        Port br-dvs
            Interface br-dvs
                type: internal
        Port "tap0898fb4e-9d"
            Interface "tap0898fb4e-9d"
                type: internal
    ovs_version: "2.6.1"

$ ip netns exec qdhcp-f2c516f0-e442-4976-86d5-b58ce5145536 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
49: tap0898fb4e-9d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:c1:68:5a brd ff:ff:ff:ff:ff:ff
    inet 10.168.75.11/24 brd 10.168.75.255 scope global tap0898fb4e-9d
       valid_lft forever preferred_lft forever
    inet 169.254.169.254/16 brd 169.254.255.255 scope global tap0898fb4e-9d
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fec1:685a/64 scope link
       valid_lft forever preferred_lft forever

$ ip netns exec qdhcp-f2c516f0-e442-4976-86d5-b58ce5145536 ping 10.168.75.11
PING 10.168.75.11 (10.168.75.11) 56(84) bytes of data.
64 bytes from 10.168.75.11: icmp_seq=1 ttl=64 time=0.035 ms
64 bytes from 10.168.75.11: icmp_seq=2 ttl=64 time=0.021 ms
^C
--- 10.168.75.11 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.021/0.028/0.035/0.007 ms

$ ip netns exec qdhcp-f2c516f0-e442-4976-86d5-b58ce5145536 ping 10.168.75.1
PING 10.168.75.1 (10.168.75.1) 56(84) bytes of data.
From 10.168.75.11 icmp_seq=1 Destination Host Unreachable
From 10.168.75.11 icmp_seq=2 Destination Host Unreachable
From 10.168.75.11 icmp_seq=3 Destination Host Unreachable
From 10.168.75.11 icmp_seq=4 Destination Host Unreachable
^C
--- 10.168.75.1 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4000ms
pipe 4

$ tail /var/log/neutron/openvswitch-agent.log
2018-01-26 13:46:29.877 22666 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-77688efb-9a24-4600-8562-3fd28796da1a - - - - -] Starting to process devices in:{'current': set([u'0898fb4e-9dfe-4994-9419-105b36dbb4aa']), 'removed': set([]), 'added': set([u'0898fb4e-9dfe-4994-9419-105b36dbb4aa'])} rpc_loop /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:2052
2018-01-26 13:46:29.878 22666 DEBUG oslo_messaging._drivers.amqpdriver [req-77688efb-9a24-4600-8562-3fd28796da1a - - - - -] CALL msg_id: ee0452ffd6e54a5db8cdb56ea5541e5a exchange 'neutron' topic 'q-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:568
2018-01-26 13:46:29.884 22666 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: ee0452ffd6e54a5db8cdb56ea5541e5a __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:416
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-77688efb-9a24-4600-8562-3fd28796da1a - - - - -] Error while processing VIF ports
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2054, in rpc_loop
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, ovs_restarted)
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return f(*args, **kwargs)
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1637, in process_network_ports
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent devices_added_updated, ovs_restarted))
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return f(*args, **kwargs)
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1510, in treat_devices_added_or_updated
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.conf.host))
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 146, in get_devices_details_list_and_failed_devices
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent devices=devices, agent_id=agent_id, host=host)
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 162, in call
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return self._original_context.call(ctxt, method, **kwargs)
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry)
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent timeout=timeout, retry=retry)
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent raise result
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent RemoteError: Remote error: NoSuchMethod Endpoint does not support RPC method get_devices_details_list_and_failed_devices
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming\n res = self.dispatcher.dispatch(message)\n', u' File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 155, in dispatch\n raise NoSuchMethod(method)\n', u'NoSuchMethod: Endpoint does not support RPC method get_devices_details_list_and_failed_devices\n'].
2018-01-26 13:46:29.884 22666 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2018-01-26 13:46:29.885 22666 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-77688efb-9a24-4600-8562-3fd28796da1a - - - - -] Agent rpc_loop - iteration:5639 completed. Processed ports statistics: {'regular': {'updated': 0, 'added': 1, 'removed': 0}}. Elapsed:0.120 loop_count_and_wait /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:1786

[youngnoh kim (zeroxkim)]

I solved it.
I changed dhcp_driver from neutron.agent.linux.dhcp.Dnsmasq to vmware_nsx.plugins.dvs.dhcp.Dnsmasq.
And I changed from ovs_integration_bridge to dvs_integration_bridge.
Neutron-openvswitch-agent is not used.

$ cat /etc/neutron/dhcp_agent.ini
[DEFAULT]
interface_driver = openvswitch
dhcp_driver = vmware_nsx.plugins.dvs.dhcp.Dnsmasq
dvs_integration_bridge = br-dvs
enable_isolated_metadata = true
enable_metadata_network = true
ovs_use_veth = true