Comment 0 for bug 1644251

Detailed bug description:
 Fuel 9.X snapshots_id: 541
Steps to reproduce:
            1. Upload plugins to the master node.
            2. Create cluster with vcenter.
            3. Add nodes with the following roles:
                * controller
                * controller
                * controller
                * compute-vmware + cinder-vmware
                * compute + cinder
            4. Configure vcenter.
            5. Configure NSXt for that cluster.
            6. Deploy cluster.
            7. Run OSTF.

Expected results:
 * OSTF should be passed successfully
 * instances should get ip addresses successfully
 * instances should be accessible via floating ip

Actual result:
 TimeoutError: OSTF tests run timeout (cluster_id=1)
 And instances can't get IP address

Reproducibility:
    100%

Impact:
 NSXT plugin under 9.2

Additional information:
 testgroup: nsxt_bvt

2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp/agent.py", line 112, in call_driver
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs)
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 208, in enable
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network)
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 1219, in setup
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent port = self.setup_dhcp_port(network)
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 1181, in setup_dhcp_port
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent dhcp_port = setup_method(network, device_id, dhcp_subnets)
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/dhcp.py", line 1160, in _setup_new_dhcp_port
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent return self.plugin.create_dhcp_port({'port': port_dict})
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/agent/dhcp/agent.py", line 455, in create_dhcp_port
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent port=port, host=self.host)
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 136, in call
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent return self._original_context.call(ctxt, method, **kwargs)
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent retry=self.retry)
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 91, in _send
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent timeout=timeout, retry=retry)
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 512, in send
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent retry=retry)
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 503, in _send
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent raise result
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent RemoteError: Remote error: IpAddressGenerationFailure No more IP addresses available on network a3f51080-4728-4a1e-839e-a28ab15a78a6.
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch\n result = func(ctxt, **new_args)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper\n ectxt.value = e.inner_exc\n', u' File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper\n return f(*args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/neutron/quota/resource_registry.py", line 92, in wrapper\n ret_val = f(_self, context, *args, **kwargs)\n', u' File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 221, in create_dhcp_port\n return self._port_action(plugin, context, port, \'create_port\')\n', u' File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 109, in _port_action\n {"action": action, "net_id": net_id, \'reason\': e})\n', u' File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__\n self.force_reraise()\n', u' File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n', u' File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 86, in _port_action\n return p_utils.create_port(plugin, context, port)\n', u' File "/usr/lib/python2.7/dist-packages/neutron/plugins/common/utils.py", line 161, in create_port\n return core_plugin.create_port(context, {\'port\': port_data})\n', u' File "/usr/lib/python2.7/dist-packages/vmware_nsx/plugins/nsx_v3/plugin.py", line 884, in create_port\n neutron_db = super(NsxV3Plugin, self).create_port(context, port)\n', u' File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1208, in create_port\n db_port = self.create_port_db(context, port)\n', u' File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1249, in create_port_db\n port_id)\n', u' File "/usr/lib/python2.7/dist-packages/neutron/db/ipam_non_pluggable_backend.py", line 204, in allocate_ips_for_port_and_store\n ips = self._allocate_ips_for_port(context, port)\n', u' File "/usr/lib/python2.7/dist-packages/neutron/db/ipam_non_pluggable_backend.py", line 361, in _allocate_ips_for_port\n p[\'mac_address\'])\n', u' File "/usr/lib/python2.7/dist-packages/neutron/db/ipam_non_pluggable_backend.py", line 310, in _allocate_fixed_ips\n result = self._generate_ip(context, subnets)\n', u' File "/usr/lib/python2.7/dist-packages/neutron/db/ipam_non_pluggable_backend.py", line 46, in _generate_ip\n return IpamNonPluggableBackend._try_generate_ip(context, subnets)\n', u' File "/usr/lib/python2.7/dist-packages/neutron/db/ipam_non_pluggable_backend.py", line 83, in _try_generate_ip\n raise n_exc.IpAddressGenerationFailure(net_id=subnets[0][\'network_id\'])\n', u'IpAddressGenerationFailure: No more IP addresses available on network a3f51080-4728-4a1e-839e-a28ab15a78a6.\n'].
2016-11-23 13:36:30.561 7776 ERROR neutron.agent.dhcp.agent

Full log of dhcp-agent:
http://paste.ubuntu.com/23522370/