Re-ran rally and saw this go by:
2017-12-07 00:02:25.399 5255 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 74 START 2017-12-07 00:02:25.915 5252 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 67 END: OK 2017-12-07 00:02:25.927 5252 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 75 START 2017-12-07 00:02:26.202 5254 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 41 END: Error ConnectFailure: Unable to establish connection to http://10.245.208.97:9696/v2.0/subnets/d6fe1572-83ca-4f64-a30e-41522471e2f9: ('Connection aborted.', BadStatusLine("''",)) 2017-12-07 00:02:26.217 5254 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 76 START 2017-12-07 00:02:26.601 5255 INFO rally.task.runner [-] Task a502ee62-c31b-4333-8169-f6a3d07d592e | ITER: 73 END: OK 2017-12-07 00:02:26.626 5255 INFO rally.task.runner [-] Task a5
BadStatusLine("''",) is the smoking gun. It is almost always haproxy dropping the connection due to one of its timeouts.
I highly recommend adding the follwowing configurations. For all the OpenStack API charms:
juju confi neutron-api haproxy-server-timeout=90000 haproxy-client-timeout=90000 haproxy-queue-timeout=9000 haproxy-connect-timeout=9000
The defaults are good for non-busy clouds. But once we are stress testing we need to bump up the timeouts so that haproxy does not drop connections. This is what we have running in serverstack.
Re-ran rally and saw this go by:
2017-12-07 00:02:25.399 5255 INFO rally.task.runner [-] Task a502ee62- c31b-4333- 8169-f6a3d07d59 2e | ITER: 74 START c31b-4333- 8169-f6a3d07d59 2e | ITER: 67 END: OK c31b-4333- 8169-f6a3d07d59 2e | ITER: 75 START c31b-4333- 8169-f6a3d07d59 2e | ITER: 41 END: Error ConnectFailure: Unable to establish connection to http:// 10.245. 208.97: 9696/v2. 0/subnets/ d6fe1572- 83ca-4f64- a30e-41522471e2 f9: ('Connection aborted.', BadStatusLine( "''",)) c31b-4333- 8169-f6a3d07d59 2e | ITER: 76 START c31b-4333- 8169-f6a3d07d59 2e | ITER: 73 END: OK
2017-12-07 00:02:25.915 5252 INFO rally.task.runner [-] Task a502ee62-
2017-12-07 00:02:25.927 5252 INFO rally.task.runner [-] Task a502ee62-
2017-12-07 00:02:26.202 5254 INFO rally.task.runner [-] Task a502ee62-
2017-12-07 00:02:26.217 5254 INFO rally.task.runner [-] Task a502ee62-
2017-12-07 00:02:26.601 5255 INFO rally.task.runner [-] Task a502ee62-
2017-12-07 00:02:26.626 5255 INFO rally.task.runner [-] Task a5
BadStatusLine( "''",) is the smoking gun. It is almost always haproxy dropping the connection due to one of its timeouts.
I highly recommend adding the follwowing configurations. For all the OpenStack API charms:
juju confi neutron-api haproxy- server- timeout= 90000 haproxy- client- timeout= 90000 haproxy- queue-timeout= 9000 haproxy- connect- timeout= 9000
The defaults are good for non-busy clouds. But once we are stress testing we need to bump up the timeouts so that haproxy does not drop connections. This is what we have running in serverstack.