Ok, just to clarify...

By 'fine' I mean that this happens as well:

After Stage1 completes, Remove the foreign check on the SQL and restart neutron-server
It will start deploying a metadata-proxy edge but throws this stackdump:

2016-10-22 09:53:37.124 20546 DEBUG oslo_concurrency.lockutils [req-1152bc36-e2e0-4ea6-a78b-2335a06389ee - - - - -] Acquired semaphore "nsx-edge-pool" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:212
2016-10-22 09:53:37.216 20546 DEBUG oslo_concurrency.lockutils [req-1152bc36-e2e0-4ea6-a78b-2335a06389ee - - - - -] Releasing semaphore "nsx-edge-pool" lock /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:225
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy [-] Exception This transaction is closed while creating internal edge for metadata service
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy Traceback (most recent call last):
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy   File "/usr/lib/python2.7/site-packages/vmware_nsx/plugins/nsx_v/md_proxy.py", line 385, in _setup_new_proxy_edge
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy     allow_metadata=False)
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy   File "/usr/lib/python2.7/site-packages/vmware_nsx/plugins/nsx_v/plugin.py", line 1569, in create_router
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy     self._process_nsx_router_create(context, router_db, r)
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 494, in __exit__
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy     self._assert_active(deactive_ok=True, prepared_ok=True)
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 223, in _assert_active
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy     raise sa_exc.ResourceClosedError(closed_msg)
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy ResourceClosedError: This transaction is closed
2016-10-22 09:53:37.315 20546 ERROR vmware_nsx.plugins.nsx_v.md_proxy 
2016-10-22 09:53:37.495 20546 DEBUG vmware_nsx.db.nsxrouter [-] Nsx router extension successfully processed for router:00e6a406-c1b9-44a9-be02-a50a49926e05 _process_nsx_router_create /usr/lib/python2.7/site-packages/vmware_nsx/db/nsxrouter.py:63





The metadaproxy is indeed deployed and in the database. Seconds later (after adding vnics and other things it seems) it dies again with:

2016-10-22 09:53:56.490 20546 ERROR neutron Traceback (most recent call last):                                                                                                                                                                                                                                        [30/1964]
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/bin/neutron-server", line 10, in <module>
2016-10-22 09:53:56.490 20546 ERROR neutron     sys.exit(main())
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/server/__init__.py", line 22, in main
2016-10-22 09:53:56.490 20546 ERROR neutron     server.boot_server(_main_neutron_server)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 37, in boot_server
2016-10-22 09:53:56.490 20546 ERROR neutron     server_func()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/server/__init__.py", line 27, in _main_neutron_server
2016-10-22 09:53:56.490 20546 ERROR neutron     wsgi_eventlet.eventlet_wsgi_server()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 23, in eventlet_wsgi_server
2016-10-22 09:53:56.490 20546 ERROR neutron     neutron_api = service.serve_wsgi(service.NeutronApiService)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 110, in serve_wsgi
2016-10-22 09:53:56.490 20546 ERROR neutron     LOG.exception(_LE('Unrecoverable error: please check log '
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-22 09:53:56.490 20546 ERROR neutron     self.force_reraise()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-22 09:53:56.490 20546 ERROR neutron     six.reraise(self.type_, self.value, self.tb)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 107, in serve_wsgi
2016-10-22 09:53:56.490 20546 ERROR neutron     service.start()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 80, in start
2016-10-22 09:53:56.490 20546 ERROR neutron     self.wsgi_app = _run_wsgi(self.app_name)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/service.py", line 234, in _run_wsgi
2016-10-22 09:53:56.490 20546 ERROR neutron     app = config.load_paste_app(app_name)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/common/config.py", line 287, in load_paste_app
2016-10-22 09:53:56.490 20546 ERROR neutron     app = loader.load_app(app_name)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_service/wsgi.py", line 353, in load_app
2016-10-22 09:53:56.490 20546 ERROR neutron     return deploy.loadapp("config:%s" % self.config_path, name=name)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in loadapp
2016-10-22 09:53:56.490 20546 ERROR neutron     return loadobj(APP, uri, name=name, **kw)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in loadobj
2016-10-22 09:53:56.490 20546 ERROR neutron     return context.create()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-10-22 09:53:56.490 20546 ERROR neutron     return self.object_type.invoke(self)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-10-22 09:53:56.490 20546 ERROR neutron     **context.local_conf)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-10-22 09:53:56.490 20546 ERROR neutron     val = callable(*args, **kw)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2016-10-22 09:53:56.490 20546 ERROR neutron     app = loader.get_app(app_name, global_conf=global_conf)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-10-22 09:53:56.490 20546 ERROR neutron     name=name, global_conf=global_conf).create()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-10-22 09:53:56.490 20546 ERROR neutron     return self.object_type.invoke(self)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-10-22 09:53:56.490 20546 ERROR neutron     **context.local_conf)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-10-22 09:53:56.490 20546 ERROR neutron     val = callable(*args, **kw)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
2016-10-22 09:53:56.490 20546 ERROR neutron     app = loader.get_app(pipeline[-1])
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in get_app
2016-10-22 09:53:56.490 20546 ERROR neutron     name=name, global_conf=global_conf).create()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-10-22 09:53:56.490 20546 ERROR neutron     return self.object_type.invoke(self)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-10-22 09:53:56.490 20546 ERROR neutron     return fix_call(context.object, context.global_conf, **context.local_conf)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-10-22 09:53:56.490 20546 ERROR neutron     val = callable(*args, **kw)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 71, in factory
2016-10-22 09:53:56.490 20546 ERROR neutron     return cls(**local_config)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 75, in __init__
2016-10-22 09:53:56.490 20546 ERROR neutron     plugin = manager.NeutronManager.get_plugin()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 233, in get_plugin
2016-10-22 09:53:56.490 20546 ERROR neutron     return weakref.proxy(cls.get_instance().plugin)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 227, in get_instance
2016-10-22 09:53:56.490 20546 ERROR neutron     cls._create_instance()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2016-10-22 09:53:56.490 20546 ERROR neutron     return f(*args, **kwargs)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 213, in _create_instance
2016-10-22 09:53:56.490 20546 ERROR neutron     cls._instance = cls()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 116, in __init__
2016-10-22 09:53:56.490 20546 ERROR neutron     plugin_provider)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/manager.py", line 149, in _get_plugin_instance
2016-10-22 09:53:56.490 20546 ERROR neutron     return plugin_class()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line 121, in wrapper
2016-10-22 09:53:56.490 20546 ERROR neutron     return f(*args, **kwargs)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/vmware_nsx/plugins/nsx_v/plugin.py", line 186, in __init__
2016-10-22 09:53:56.490 20546 ERROR neutron     nsx_v_md_proxy.NsxVMetadataProxyHandler(self))
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/vmware_nsx/plugins/nsx_v/md_proxy.py", line 116, in __init__
2016-10-22 09:53:56.490 20546 ERROR neutron     self.proxy_edge_ips = self._get_proxy_edges()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/vmware_nsx/plugins/nsx_v/md_proxy.py", line 265, in _get_proxy_edges
2016-10-22 09:53:56.490 20546 ERROR neutron     self._setup_new_proxy_edge, remaining_cfg_ips):
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/greenpool.py", line 238, in next
2016-10-22 09:53:56.490 20546 ERROR neutron     val = self.waiters.get().wait()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 175, in wait
2016-10-22 09:53:56.490 20546 ERROR neutron     return self._exit_event.wait()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 125, in wait
2016-10-22 09:53:56.490 20546 ERROR neutron     current.throw(*self._exc)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
2016-10-22 09:53:56.490 20546 ERROR neutron     result = function(*args, **kwargs)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/vmware_nsx/plugins/nsx_v/md_proxy.py", line 458, in _setup_new_proxy_edge
2016-10-22 09:53:56.490 20546 ERROR neutron     self.context, filters={'device_id': [rtr_id]})
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib/python2.7/site-packages/neutron/db/db_base_plugin_v2.py", line 1387, in get_ports
2016-10-22 09:53:56.490 20546 ERROR neutron     for c in query:
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
2016-10-22 09:53:56.490 20546 ERROR neutron     return self._execute_and_instances(context)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
2016-10-22 09:53:56.490 20546 ERROR neutron     close_with_result=True)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
2016-10-22 09:53:56.490 20546 ERROR neutron     **kw)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 905, in connection
2016-10-22 09:53:56.490 20546 ERROR neutron     execution_options=execution_options)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind
2016-10-22 09:53:56.490 20546 ERROR neutron     engine, execution_options)
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 313, in _connection_for_bind
2016-10-22 09:53:56.490 20546 ERROR neutron     self._assert_active()
2016-10-22 09:53:56.490 20546 ERROR neutron   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 202, in _assert_active
2016-10-22 09:53:56.490 20546 ERROR neutron     "This session is in 'prepared' state; no further "
2016-10-22 09:53:56.490 20546 ERROR neutron InvalidRequestError: This session is in 'prepared' state; no further SQL can be emitted within this transaction.
2016-10-22 09:53:56.490 20546 ERROR neutron 


After waiting 60s (One Coffee) and restarting the neutron process it starts up fine. Yet it spawns another metadata_proxy edge.

Both edges have the same IP assigned.

After this back and forth I delete the first metadataproxy from the SQL and destroy it's edge as I assume it's 'broken'.

The proxy itself works as expected. The deployment however is very tricky it seems