[VMware] Time to time creating instance has failed during performance testing

[Olesia Tsvigun]

Fuel ISO #525
OS Ubuntu
Fuel vmware-dvs plugin(http://jenkins-tpi.bud.mirantis.net:8080/job/fuel-plugin.dvs.build/lastSuccessfulBuild/)

Description
Time to Time creating instance has failed during perfomane testing.

Test : http://mos-scale.vm.mirantis.net/test_results/build_6.1-525/jenkins-13_env_run_rally_light-116-ubuntu-MSK-2015-09-09-10:43:14-2015-09-09-16:16:51/rally_report.html#/NovaServers.boot_and_delete_server/failures

From rally logs
http://paste.openstack.org/show/454850/

Reproducibility: 5.1 %

[Andrzej Skupień (kendriu)]

I have committed today some changes that should fix this problem.

I'm Closing this bug. Please reopen if it occur again.

[Vyacheslav Tabolin (slavchick)]

I suppose to find out the reason in vSphere - 'Cannot complete operation due to concurrent modification by another operation'.

Neutron server logs needs.

[Andrzej Skupień (kendriu)]

'Cannot complete operation due to concurrent modification by another operation' is normal communicate, when race condition occurs. In this case operation is repeated (decorator @wrap_retry).

It's normal situation in vmware driver operation flow and it doesn't cause any errors.

[Andrzej Skupień (kendriu)]

If you change some properties of PortGroup for example, then you first check configVersion of it current config. Then you put this version in configVersion field of ReconfigureDVPortgroup_Task communicate. DVS first checks if configVersion in received ReconfigureDVPortgroup_Task is the same as configVersion of current PortGroup state. If they are the same then DVS applies changes, if not then DVS "throws" exception "Cannot complete operation due to concurrent modification...", because this means that other, parallel operation already changed state of PG. In this case driver have to verify current state of PG and send (or not) his changes again.

[Vyacheslav Tabolin (slavchick)]

But @wrap_retry operation repeater may cause timeout problems on other modules. I can't deploy more then 4 VM's at the same time on lab correctly.

[Andrzej Skupień (kendriu)]

that's true, but I didn't found other solution except increasing timeout in other modules, such as haproxy and nova-compute.

[Olesia Tsvigun (otsvigun)]

Can not reproduced anymore at fuel-plugin-vmware-dvs 7.0
+----------------------------------------------------------------------------------------------+
| Response Times (sec) |
+--------------------+--------+--------+--------+---------+---------+--------+---------+-------+
| action | min | median | 90%ile | 95%ile | max | avg | success | count |
+--------------------+--------+--------+--------+---------+---------+--------+---------+-------+
| nova.boot_server | 28.624 | 37.951 | 55.442 | 68.896 | 236.14 | 43.365 | 99.9% | 1311 |
| nova.delete_server | 13.128 | 19.987 | 34.674 | 43.093 | 277.183 | 24.527 | 99.9% | 1311 |
| total | 47.823 | 60.865 | 87.136 | 107.421 | 310.639 | 67.892 | 99.9% | 1311 |
+--------------------+--------+--------+--------+---------+---------+--------+---------+-------+
Load duration: 4508.07324696
Full duration: 4587.78851795