Steps:
Scenario:
1. Destroy first controller
2. Check pacemaker status
3. Run OSTF
4. Revert environment
5. Destroy second controller
6. Check pacemaker status
7. Run OSTF
Actual Result:
1. OSTF from step 3 failed with:
- Check network connectivity from instance via floating IP (failure) Failed to create rule in security group. Please refer to OpenStack logs for more details.
In ostf log we can see that test failed in 00:51 by timeout:
00:51:00 FAILURE Check network connectivity from instance via floating IP
File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
httplib_response = conn.getresponse(buffering=True)
File "/usr/lib64/python2.7/httplib.py", line 1045, in getresponse
response.begin()
File "/usr/lib64/python2.7/httplib.py", line 409, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib64/python2.7/socket.py", line 476, in readline
data = self._sock.recv(self._rbufsize)
File "/usr/lib64/python2.7/ssl.py", line 241, in recv
return self.read(buflen)
File "/usr/lib64/python2.7/ssl.py", line 160, in read
return self._sslobj.read(len)
File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 194, in _raise_TimeOut
raise TimeOutError()
TimeOutError
At the same time, in nova log we can see that exactly in 00:51 rule was added, but we interupt the test and send delete request:
016-01-11T00:51:09.192093+00:00 debug: 2016-01-11 00:51:09.190 17855 DEBUG keystoneclient.session [req-80d18190-cb98-41d1-aeeb-f2ec9e698a72 0abb6a958ef64155992546a567b8665f 6d52f3fd0d314df88dfe3f69c2e33375 - - -] RESP: [200] date: Mon, 11 Jan 2016 00:51:09 GMT connection: close content-type: application/json; charset=UTF-8 content-length: 1224 x-openstack-request-id: req-9569a914-15f5-4625-b3d1-3a82fe922ee5
RESP BODY: {"security_group": {"tenant_id": "6d52f3fd0d314df88dfe3f69c2e33375", "description": "ost1_test-secgroup-smoke-netw1597447401 description", "id": "e72f8469-3475-4954-bc63-5065bb4da63c", "security_group_rules": [{"remote_group_id": null, "direction": "egress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv6", "tenant_id": "6d52f3fd0d314df88dfe3f69c2e33375", "port_range_max": null, "port_range_min": null, "id": "1c65e0bd-5076-40b6-8997-784f24183caa", "security_group_id": "e72f8469-3475-4954-bc63-5065bb4da63c"}, {"remote_group_id": null, "direction": "egress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv4", "tenant_id": "6d52f3fd0d314df88dfe3f69c2e33375", "port_range_max": null, "port_range_min": null, "id": "c2c4069a-bb38-4de1-9cc6-21d0d31f1de5", "security_group_id": "e72f8469-3475-4954-bc63-5065bb4da63c"}, {"remote_group_id": null, "direction": "ingress", "remote_ip_prefix": "0.0.0.0/0", "protocol": "tcp", "ethertype": "IPv4", "tenant_id": "6d52f3fd0d314df88dfe3f69c2e33375", "port_range_max": 22, "port_range_min": 22, "id": "fa670538-f41f-48a3-8a9a-8176d7e67356", "security_group_id": "e72f8469-3475-4954-bc63-5065bb4da63c"}], "name": "ost1_test-secgroup-smoke-netw1597447401"}}
_http_log_response /usr/lib/python2.7/dist-packages/keystoneclient/session.py:215
2016-01-11T00:51:09.194155+00:00 debug: 2016-01-11 00:51:09.192 17855 DEBUG keystoneclient.session [req-80d18190-cb98-41d1-aeeb-f2ec9e698a72 0abb6a958ef64155992546a567b8665f 6d52f3fd0d314df88dfe3f69c2e33375 - - -] REQ: curl -g -i -X DELETE http://10.109.11.3:9696/v2.0/security-groups/e72f8469-3475-4954-bc63-5065bb4da63c.json -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}83371d59a49e73a7f1f63646c917f5b2b03db2ae" _http_log_request /usr/lib/python2.7/dist-packages/keystoneclient/session.py:198
2016-01-11T00:51:09.324869+00:00 debug: 2016-01-11 00:51:09.320 17855 DEBUG keystoneclient.session [req-80d18190-cb98-41d1-aeeb-f2ec9e698a72 0abb6a958ef64155992546a567b8665f 6d52f3fd0d314df88dfe3f69c2e33375 - - -] RESP: [204] date: Mon, 11 Jan 2016 00:51:09 GMT connection: close content-length: 0 x-openstack-request-id: req-38c1ccad-943e-427c-a654-c2094939987c _http_log_response /usr/lib/python2.7/dist-packages/keystoneclient/session.py:215
2016-01-11T00:51:09.325368+00:00 info: 2016-01-11 00:51:09.323 17855 INFO nova.osapi_compute.wsgi.server [req-80d18190-cb98-41d1-aeeb-f2ec9e698a72 0abb6a958ef64155992546a567b8665f 6d52f3fd0d314df88dfe3f69c2e33375 - - -] 10.109.11.3 "DELETE /v2/6d52f3fd0d314df88dfe3f69c2e33375/os-security-groups/e72f8469-3475-4954-bc63-5065bb4da63c HTTP/1.1" status: 202 len: 196 time: 2.9439869
VERSION:
feature_groups:
- mirantis
production: "docker"
release: "8.0"
api: "1.0"
build_number: "417"
build_id: "417"
fuel-nailgun_sha: "9ebbaa0473effafa5adee40270da96acf9c7d58a"
python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
fuel-library_sha: "7ef751bdc0e4601310e85b8bf713a62ed4aee305"
fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
fuelmenu_sha: "2a0def56276f0fc30fd949605eeefc43e5d7cc6c"
shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
fuelmain_sha: "cfeadd34d8d048deeabf0884931708b1d040b8a6"
ISO fuel-8. 0-574-2016- 02-16_13- 42-00.iso:
Probably main failure is generated by too fast OSTF start:
Failed tests only:
1. start :
Create volume and boot instance from it failure
Check state of haproxy backends on controllers failure
Check that required services are running failure
Check network connectivity from instance via floating IP failure
2 and more:
Check that required services are running failure
Check that required services are running failure
Seeking for safe way to pause, until cluster will return to the working state.