NSX|V: Failed to allocate one available vnic

[Gary Kotton]

Rally:
Create 8 non-overlapping networks with single subnet in each.
Create shared router and attach 8 networks to the router.
Launch single instance in each networks(Total 8VM’s/8Networks].
Repeat this same step(1-3) on 125 tenants with concurrency of 10.

Rally: http://10.112.17.44/rally/10conc_shared_nw_scale_aggreagte_fix.html
VIO tech-support attached to bug.

Logs:

2015-09-16 13:04:50.523 16082 DEBUG oslo_concurrency.lockutils [req-ae222114-d9a7-410d-824e-ead8411ab0bc ] Releasing file lock "/var/lib/neutron/lock/bind-router" after holding it for 55.721s release /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:227
2015-09-16 13:04:50.525 16082 DEBUG oslo_concurrency.lockutils [req-ae222114-d9a7-410d-824e-ead8411ab0bc ] Releasing semaphore "router" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:\404
2015-09-16 13:04:50.526 16082 ERROR neutron.api.v2.resource [req-ae222114-d9a7-410d-824e-ead8411ab0bc ] add_router_interface failed
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource Traceback (most recent call last):
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/resource.py", line 83, in resource
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource result = method(request=request, **args)
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py", line 207, in _handle_action
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource return getattr(self._plugin, name)(*arg_list, **kwargs)
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/plugins/nsx_v.py", line 1633, in add_router_interface
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource context, router_id, interface_info)
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/plugins/nsx_v_drivers/shared_router_driver.py", line 709, in add_router_interface
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource context, router_id, router_db.admin_state_up)
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/plugins/nsx_v_drivers/shared_router_driver.py", line 492, in _bind_router_on_available_edge
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource address_groups, admin_state)
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/vshield/edge_utils.py", line 1564, in update_internal_interface
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource context.session, edge_id, int_net_id)
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/dbexts/nsxv_db.py", line 185, in allocate_edge_vnic
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource raise nsx_exc.NsxPluginException(err_msg=msg)
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource NsxPluginException: An unexpected error occurred in the NSX Plugin: Failed to allocate one available vnic on edge_id: :edge-151 to network_id: e4c99de6-6ccb-4875-ac68-b89efc8277a4
2015-09-16 13:04:50.526 16082 TRACE neutron.api.v2.resource
2015-09-16 13:04:50.532 16080 DEBUG oslo_concurrency.lockutils [req-073c82f9-02b1-4f31-8ada-b91166b30634 ] Acquired file lock "/var/lib/neutron/lock/bind-router" after waiting 145.655s acquire /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/249620
Committed: https://git.openstack.org/cgit/openstack/vmware-nsx/commit/?id=6f2b2976121246bb6413d2e789087dc4daf0f29d
Submitter: Jenkins
Branch: master

commit 6f2b2976121246bb6413d2e789087dc4daf0f29d
Author: Gary Kotton <email address hidden>
Date: Sun Nov 22 00:22:39 2015 -0800

    NSX|V: add locking for interface management

    There are some cases where this was not locked. This patch ensures
    that it is done in a more unified manner.

    In addition to this the patch adds additional locking when the
    conflicting shared routers are check. This is to ensure that the
    router interface is taken into account.

    Co-Authored-By: Bo Lin <email address hidden>

    Change-Id: Ib760b0bf5fd7341e04c2116d69f7c39c17331413
    Closes-bug: #1519667