NSXv: operation hangs when creating a router

[Gary Kotton]

When an edge pool is exhausted and a router is created the operation would hang

2015-08-19 04:13:40.153 623 DEBUG oslo_messaging._drivers.amqp [req-2789d27d-be1c-45cf-865a-8e347e9f1054 ] UNIQUE_ID is af07bfa36d6248babdc0111a91f083d6. _add_unique_id /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py:252
2015-08-19 04:13:40.167 623 DEBUG neutron.api.v2.base [req-2789d27d-be1c-45cf-865a-8e347e9f1054 ] Request body: {u'router': {u'router_type': u'exclusive', u'name': u'sample2', u'admin_state_up': True}} prepare_request_body /usr/lib/python2.7/dist-packages/neutron/api/v2/base.py:597
2015-08-19 04:13:40.233 623 DEBUG vmware_nsx.neutron.plugins.vmware.dbexts.nsxrouter [req-2789d27d-be1c-45cf-865a-8e347e9f1054 ] Nsx router extension successfully processed for router:c900e31e-bd9c-4da9-aa8f-7fb6f96be7d5 _process_nsx_router_create /usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/dbexts/nsxrouter.py:62
2015-08-19 04:13:40.239 623 DEBUG oslo_concurrency.lockutils [req-2789d27d-be1c-45cf-865a-8e347e9f1054 ] Acquired semaphore "nsx-edge-request" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:391
2015-08-19 04:13:40.240 623 DEBUG oslo_concurrency.lockutils [req-2789d27d-be1c-45cf-865a-8e347e9f1054 ] Acquired file lock "/var/lib/neutron/lock/get-nsx-edge-request" after waiting 0.000s acquire /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:212
2015-08-19 04:13:40.250 623 DEBUG oslo_concurrency.lockutils [req-2789d27d-be1c-45cf-865a-8e347e9f1054 ] Releasing file lock "/var/lib/neutron/lock/get-nsx-edge-request" after holding it for 0.010s release /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:227
2015-08-19 04:13:40.251 623 DEBUG oslo_concurrency.lockutils [req-2789d27d-be1c-45cf-865a-8e347e9f1054 ] Releasing semaphore "nsx-edge-request" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:404
2015-08-19 04:13:40.257 623 DEBUG vmware_nsx.neutron.plugins.vmware.vshield.tasks.tasks [-] Start task Task-deploying-sample2-c900e31e-bd9c-4da9-aa8f-7fb6f96be7d5-c900e31e-bd9c-4da9-aa8f-7fb6f96be7d5-abf9b784-4628-11e5-9f17-005056bc8a96 _execute /usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/vshield/tasks/tasks.py:183
2015-08-19 04:13:40.258 623 DEBUG vmware_nsx.neutron.plugins.vmware.vshield.edge_appliance_driver [-] NSXv: start deploying edge _deploy_edge /usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/vshield/edge_appliance_driver.py:361
2015-08-19 04:13:40.258 623 DEBUG vmware_nsx.neutron.plugins.vmware.vshield.vcns [-] VcnsApiHelper('POST', '/api/4.0/edges?async=true', '{"appliances": {"appliances": [{"resourcePoolId": "resgroup-236", "datastoreId": "datastore-58"}], "applianceSize": "compact"}, "name": "sample2-c900e31e-bd9c-4da9-aa8f-7fb6f96be7d5", "enableFips": false, "autoConfiguration": {"rulePriority": "high", "enabled": false}, "fqdn": null, "datacenterMoid": "datacenter-2", "featureConfigs": {"features": [{"featureType": "loadbalancer_4.0", "enabled": true}]}, "cliSettings": {"remoteAccess": true}, "vnics": {"vnics": [{"enableProxyArp": false, "index": 0, "enableSendRedirects": true, "name": "external", "isConnected": true, "subInterfaces": {"subInterfaces": []}, "type": "uplink", "portgroupId": "dvportgroup-78", "mtu": 1500}]}, "type": "gatewayServices", "enableAesni": true}') do_request /usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/vshield/vcns.py:93
2015-08-19 04:13:40.557 623 DEBUG vmware_nsx.neutron.plugins.vmware.vshield.vcns [-] VcnsApiHelper('GET', '/api/4.0/edges/jobs/jobdata-2149', 'null') do_request /usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/vshield/vcns.py:93
2015-08-19 04:13:40.638 623 DEBUG vmware_nsx.neutron.plugins.vmware.vshield.edge_appliance_driver [-] VCNS: deploying edge edge-76 _deploy_edge /usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/vshield/edge_appliance_driver.py:369
2015-08-19 04:13:40.639 623 DEBUG vmware_nsx.neutron.plugins.vmware.vshield.tasks.tasks [-] Task Task-deploying-sample2-c900e31e-bd9c-4da9-aa8f-7fb6f96be7d5-c900e31e-bd9c-4da9-aa8f-7fb6f96be7d5-abf9b784-4628-11e5-9f17-005056bc8a96 return 1 _execute /usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/vshield/tasks/tasks.py:196
2015-08-19 04:13:40.639 623 DEBUG vmware_nsx.neutron.plugins.vmware.vshield.edge_utils [-] Start deploying edge-76 for router sample2-c900e31e-bd9c-4da9-aa8f-7fb6f96be7d5 edge_deploy_started_sync /usr/local/lib/python2.7/dist-packages/vmware_nsx/neutron/plugins/vmware/vshield/edge_utils.py:1711

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/215273
Committed: https://git.openstack.org/cgit/openstack/vmware-nsx/commit/?id=d742705aceef78b024a34c508233d608e2e47291
Submitter: Jenkins
Branch: master

commit d742705aceef78b024a34c508233d608e2e47291
Author: Gary Kotton <email address hidden>
Date: Thu Aug 20 09:19:19 2015 -0700

    NSXv: ensure per process task manager

    When a metadata proxy is created the task manager will be created.
    This will cause corruption of the per process task manager.

    Change-Id: I9e281c26ca48d83d6b82e5b627b5dd7041171e1b
    Closes-bug: #1487167

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/217837

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/217837
Committed: https://git.openstack.org/cgit/openstack/vmware-nsx/commit/?id=490a1dce4cb7d81c98ec579923ca82a29e12b43d
Submitter: Jenkins
Branch: stable/kilo

commit 490a1dce4cb7d81c98ec579923ca82a29e12b43d
Author: Gary Kotton <email address hidden>
Date: Thu Aug 20 09:19:19 2015 -0700

    NSXv: ensure per process task manager

    When a metadata proxy is created the task manager will be created.
    This will cause corruption of the per process task manager.

    Change-Id: I9e281c26ca48d83d6b82e5b627b5dd7041171e1b
    Closes-bug: #1487167
    (cherry picked from commit d742705aceef78b024a34c508233d608e2e47291)