Agents not joining/ losing agents when executing scenarios with more number of guests

[Sai Sindhur Malleni]

When executing tests with a large number of master-slave pairs (>=10), agents fail to join/ are lost leading to error. Hardware performance was verified to make sure that the hardware wasn't a bottleneck. Agent join timeout and loss timeout were also increased but to no use.

The scenario was: (Image-CentOS7)

description:
  This scenario launches several pairs of VMs on the same compute node.
  VM are plugged into the same private network. Useful for testing performance
  degradation when the number of VMs grows.

deployment:
  template: l2.hot
  accommodation: [pair, single_room, density: 20, compute_nodes: 2]

execution:
  tests:
  -
    title: Iperf TCP
    class: iperf_graph
    time: 60
  -
    title: Iperf UDP
    class: iperf
    udp: 1
    threads: 8
    bandwidth: 1000M

The debug log shows that several agents were lost.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/199931
Committed: https://git.openstack.org/cgit/stackforge/shaker/commit/?id=760bcfc3a22ee52aae02463d9748d4591ff0e70a
Submitter: Jenkins
Branch: master

commit 760bcfc3a22ee52aae02463d9748d4591ff0e70a
Author: Ilya Shakhat <email address hidden>
Date: Thu Jul 9 10:41:05 2015 +0300

    Split out UDP tests

    UDP tests produce significant load on controllers, they should
    be configured accordingly to scale of the cloud. This patch reduces
    amount of UDP traffic in stock scenarios, tests with old values are
    extracted to udp_* files.

    Partial bug 1470892

    Change-Id: I410bb3be0f7e267cd4aba4e95a3759e9ad6bc5c0