[9.2][nsxt] an instance can't get an ip address from the dhcp-agent

[Ruslan Khozinov]

Detailed bug description:
 Fuel 9.X snapshots_id: 541
Steps to reproduce:
            1. Upload plugins to the master node.
            2. Create cluster with vcenter.
            3. Add nodes with the following roles:
                * controller
                * controller
                * controller
                * compute-vmware + cinder-vmware
                * compute + cinder
            4. Configure vcenter.
            5. Configure NSXt for that cluster.
            6. Deploy cluster.
            7. Run OSTF.

Expected results:
 * OSTF should be passed successfully
 * instances should get ip addresses successfully
 * instances should be accessible via floating ip

Actual result:
 TimeoutError: OSTF tests run timeout (cluster_id=1)
 And instances can't get IP address

Reproducibility:
    100%

Impact:
 NSXT plugin under 9.2

Additional information:
 testgroup: nsxt_bvt

Full log of dhcp-agent:
http://paste.ubuntu.com/23522370/

[Ruslan Khozinov (rkhozinov)]

neutron.agent.dhcp.agent

Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp/agent.py", line 112, in call_driver
     getattr(driver, action)(**action_kwargs)
   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 208, in enable
     interface_name = self.device_manager.setup(self.network)
   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 1219, in setup
     port = self.setup_dhcp_port(network)
   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 1181, in setup_dhcp_port
     dhcp_port = setup_method(network, device_id, dhcp_subnets)
   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 1160, in _setup_new_dhcp_port
     return self.plugin.create_dhcp_port({'port': port_dict})
   File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp/agent.py", line 455, in create_dhcp_port
     port=port, host=self.host)
   File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 136, in call
     return self._original_context.call(ctxt, method, **kwargs)
   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
     retry=self.retry)
   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 91, in _send
     timeout=timeout, retry=retry)
   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 512, in send
     retry=retry)
   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 503, in _send
     raise result
RemoteError: Remote error: IpAddressGenerationFailure No more IP addresses available on network a3f51080-4728-4a1e-839e-a28ab15a78a6.

[Ruslan Khozinov (rkhozinov)]

2016-11-23 10:37:12.745 27354 DEBUG vmware_nsx.common.locking [-] Lock nsxv3_nsgroup_manager_init taken with traceback [('/usr/bin/neutron-server', 10, '<module>', 'sys.exit(main())'), ('/usr/lib/python2.7/dist-packages/neutron/cmd/eventlet/server/__init__.py', 22, 'main', 'server.boot_server(_main_neutron_server)'), ('/usr/lib/python2.7/dist-packages/neutron/server/__init__.py', 37, 'boot_server', 'server_func()'), ('/usr/lib/python2.7/dist-packages/neutron/cmd/eventlet/server/__init__.py', 27, '_main_neutron_server', 'wsgi_eventlet.eventlet_wsgi_server()'), ('/usr/lib/python2.7/dist-packages/neutron/server/wsgi_eventlet.py', 24, 'eventlet_wsgi_server', 'neutron_api = service.serve_wsgi(service.NeutronApiService)'), ('/usr/lib/python2.7/dist-packages/neutron/service.py', 107, 'serve_wsgi', 'service.start()'), ('/usr/lib/python2.7/dist-packages/neutron/service.py', 80, 'start', 'self.wsgi_app = _run_wsgi(self.app_name)'), ('/usr/lib/python2.7/dist-packages/neutron/service.py', 234, '_run_wsgi', 'app = config.load_paste_app(app_name)'), ('/usr/lib/python2.7/dist-packages/neutron/common/config.py', 287, 'load_paste_app', 'app = loader.load_app(app_name)'), ('/usr/lib/python2.7/dist-packages/oslo_service/wsgi.py', 353, 'load_app', 'return deploy.loadapp("config:%s" % self.config_path, name=name)'), ('/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py', 247, 'loadapp', 'return loadobj(APP, uri, name=name, **kw)'), ('/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py', 272, 'loadobj', 'return context.create()'), ('/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py', 710, 'create', 'return self.object_type.invoke(self)'), ('/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py', 144, 'invoke', '**context.local_conf)'), ('/usr/lib/python2.7/dist-packages/paste/deploy/util.py', 55, 'fix_call', 'val = callable(*args, **kw)'), ('/usr/lib/python2.7/dist-packages/paste/urlmap.py', 28, 'urlmap_factory', 'app = loader.get_app(app_name, global_conf=global_conf)'), ('/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py', 350, 'get_app', 'name=name, global_conf=global_conf).create()'), ('/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py', 710, 'create', 'return self.object_type.invoke(self)'), ('/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py', 144, 'invoke', '**context.local_conf)'), ('/usr/lib/python2.7/dist-packages/paste/deploy/util.py', 55, 'fix_call', 'val = callable(*args, **kw)'), ('/usr/lib/python2.7/dist-packages/neutron/auth.py', 71, 'pipeline_factory', 'app = loader.get_app(pipeline[-1])'), ('/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py', 350, 'get_app', 'name=name, global_conf=global_conf).create()'), ('/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py', 710, 'create', 'return self.object_type.invoke(self)'), ('/usr/lib/python2.7/dist-packages/paste/deploy/loadwsgi.py', 146, 'invoke', 'return fix_call(context.object, context.global_conf, **context.local_conf)'), ('/usr/lib/python2.7/dist-packages/paste/deploy/util.py', 55, 'fix_call', 'val = callable(*args, **kw)'), ('/usr/lib/python2.7/dist-packages/neutron/api/v2/router.py', 71, 'factory', 'return cls(**local_config)'), ('/usr...