Tacker-server got crashed for some operation like parallel creation or deletion of VNFs and updation with wrong values

[Manik Bindlish]

Release: Stable Newton
Ubuntu 16.04
Environment for NFV

Scenario1:
1.) Created VNF
2.) Trying to update VNF with it's change in disk size
3.) Tacker-server gets crashed

Error:
tacker vnf-list
Unable to establish connection to http://<node tacker side>:9890/v1.0/vnfs.json

Snippet: http://paste.openstack.org/show/618813/

Time After 14.02:
Around: 2017-08-19 14:05:36

Scenario2:
Run 100 of VNF creation/deletion parallel. Again tacker-server got crashed.
Time around: 2017-08-19 16:34

Snippet: http://paste.openstack.org/show/618816/
Scenario3:
1.) Total existing VNFs at this time is 26
2.) Start updating 100 VNFs(24 which were existing and 76 which were not existing)
3.) VNFs which were existing, updated (Pending_Update)
4.) Problem: Tacker-server got crashed again.
Time Around: 2017-08-20 02:34

[dharmendra (dharmendra-kushwaha)]

I tried to reproduce on master(rocky) with small number of VNFs(i.e. 20), but could not be able to reproduce it. I will try to verify it with same number of VNFs(i.e. 100) on master again.

[Deepak Mourya (mourya007)]

@dharmendra

Hi, can you elaborate how can i reproduce this bug after setting up the tacker environment.?

[dharmendra (dharmendra-kushwaha)]

Verified on Stable/Rocky.
No server crash observed, even in multiple iteration.

[dharmendra (dharmendra-kushwaha)]

Branch stable/Rocky

Test & observations:

* Created vnfd with samples/tosca-templates/vnfd/tosca-vnfd-hello-world.yaml
* Initiated 100+ vnf-create request in parallel.

* Most of the VNFs deployment failed with "Internal error" and VNFs went in Error state due to the timeout error in heat.

* Although in nova side, VDUs(i.e. VMs) got created(in running state).
* It might be due to limited resources.

* No issue observed in Tacker behavior.