Comment 4 for bug 1736171

Re-ran rally and saw this go by:

2017-12-07 00:02:25.399 5255 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 74 START
2017-12-07 00:02:25.915 5252 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 67 END: OK
2017-12-07 00:02:25.927 5252 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 75 START
2017-12-07 00:02:26.202 5254 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 41 END: Error ConnectFailure: Unable to establish connection to http://10.245.208.97:9696/v2.0/subnets/d6fe1572-83ca-4f64-a30e-41522471e2f9: ('Connection aborted.', BadStatusLine("''",))
2017-12-07 00:02:26.217 5254 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 76 START
2017-12-07 00:02:26.601 5255 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 73 END: OK
2017-12-07 00:02:26.626 5255 INFO rally.task.runner [-] Task a5

BadStatusLine("''",) is the smoking gun. It is almost always haproxy dropping the connection due to one of its timeouts.

I highly recommend adding the follwowing configurations. For all the OpenStack API charms:

juju confi neutron-api haproxy-server-timeout=90000 haproxy-client-timeout=90000 haproxy-queue-timeout=9000 haproxy-connect-timeout=9000

The defaults are good for non-busy clouds. But once we are stress testing we need to bump up the timeouts so that haproxy does not drop connections. This is what we have running in serverstack.