TestLoadBalancerBasic fails in Tempest

[Daisuke Morita]

2014-06-05 01:28:56.830 | {0} tempest.scenario.test_load_balancer_basic.TestLoadBalancerBasic.test_load_balancer_basic [48.722497s] ... FAILED
2014-06-05 01:28:56.831 |
2014-06-05 01:28:56.831 | Captured traceback:
2014-06-05 01:28:56.831 | ~~~~~~~~~~~~~~~~~~~
2014-06-05 01:28:56.831 | Traceback (most recent call last):
2014-06-05 01:28:56.831 | File "tempest/test.py", line 126, in wrapper
2014-06-05 01:28:56.831 | return f(self, *func_args, **func_kwargs)
2014-06-05 01:28:56.831 | File "tempest/scenario/test_load_balancer_basic.py", line 305, in test_load_balancer_basic
2014-06-05 01:28:56.831 | self._check_load_balancing()
2014-06-05 01:28:56.832 | File "tempest/scenario/test_load_balancer_basic.py", line 279, in _check_load_balancing
2014-06-05 01:28:56.832 | self._check_connection(self.vip_ip)
2014-06-05 01:28:56.832 | File "tempest/scenario/test_load_balancer_basic.py", line 196, in _check_connection
2014-06-05 01:28:56.832 | while not try_connect(check_ip, port):
2014-06-05 01:28:56.832 | File "tempest/scenario/test_load_balancer_basic.py", line 188, in try_connect
2014-06-05 01:28:56.832 | resp = urllib2.urlopen("http://{0}:{1}/".format(ip, port))
2014-06-05 01:28:56.832 | File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
2014-06-05 01:28:56.833 | return _opener.open(url, data, timeout)
2014-06-05 01:28:56.833 | File "/usr/lib/python2.7/urllib2.py", line 400, in open
2014-06-05 01:28:56.833 | response = self._open(req, data)
2014-06-05 01:28:56.833 | File "/usr/lib/python2.7/urllib2.py", line 418, in _open
2014-06-05 01:28:56.833 | '_open', req)
2014-06-05 01:28:56.833 | File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
2014-06-05 01:28:56.834 | result = func(*args)
2014-06-05 01:28:56.834 | File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
2014-06-05 01:28:56.834 | return self.do_open(httplib.HTTPConnection, req)
2014-06-05 01:28:56.834 | File "/usr/lib/python2.7/urllib2.py", line 1180, in do_open
2014-06-05 01:28:56.834 | r = h.getresponse(buffering=True)
2014-06-05 01:28:56.834 | File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
2014-06-05 01:28:56.835 | response.begin()
2014-06-05 01:28:56.835 | File "/usr/lib/python2.7/httplib.py", line 407, in begin
2014-06-05 01:28:56.835 | version, status, reason = self._read_status()
2014-06-05 01:28:56.835 | File "/usr/lib/python2.7/httplib.py", line 371, in _read_status
2014-06-05 01:28:56.835 | raise BadStatusLine(line)
2014-06-05 01:28:56.835 | BadStatusLine: ''

[Elena Ezhova (eezhova)]

This exception may be raised due to special aspects of haproxy's internal behavior. Getting it in the check_connection method means that the connection was established. That's why if we get this exception we need to return True.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/98065

[Attila Fazekas (afazekas)]

HA-Proxy version 1.4.25 (available in f20 as well) always expects he will close the connection first,
however it not MUST be true with all kind of headers and HTTP version.

If the `nc web server` started in similar way:
busybox nc -ll -p 9002 -e sh -c "echo -ne 'HTTP/1.1 200 OK\r\nContent-Length: 8\r\nConnection: close\r\nContent-Type: text/html; charset=UTF-8\r\n\r\nserver2\n'; cat >/dev/null" &
lets the haproxy to close the connection first.

HA-Proxy 1.5.0 works better (compiled with TARGET=linux2628), when the server closes the connection immediately after the response served.

I suggest to use "echo 'myhttpresponse '; cat >/dev/null" as server command, it works well with both version of haproxy.

No left-over cat processes detected.

The connection expected to be closed by all proxy/client when they receive a response with correct Content-length header and a 'Connection: close' header.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/98065
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=0c8e329c54fe00dc18309ae008e8c34c7f100ff9
Submitter: Jenkins
Branch: master

commit 0c8e329c54fe00dc18309ae008e8c34c7f100ff9
Author: Elena Ezhova <email address hidden>
Date: Thu Jun 5 12:15:39 2014 +0400

    Correct nc command in test_load_balancer_basic

    HA-Proxy always expects it will close the connection first.
    To make a backend respect it, the command that starts
    the netcat "web server" should be modified by adding
    'cat >/dev/null' to a responce. Thus a client is made to wait
    until the proxy closes the connection.

    Closes bug: 1326607

    Change-Id: I762f7b59bd1fd76471cf5530882b6b08e8c9bb0b