Fuel for OpenStack

Add security rule step failed by timeout after destructive action

  1. Series newton
  2. Bug #1532874
Bug #1532874 reported by Tatyanka
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Confirmed
Medium
Fuel QA Team
Fuel for OpenStack 10.0
Mitaka
Won't Fix
Medium
Fuel QA Team
Fuel for OpenStack 9.0
Newton
Confirmed
Medium
Fuel QA Team
Fuel for OpenStack 10.0

Bug Description

Steps:
Scenario:
1. Destroy first controller
2. Check pacemaker status
3. Run OSTF
4. Revert environment
5. Destroy second controller
6. Check pacemaker status
7. Run OSTF

Actual Result:
1. OSTF from step 3 failed with:
 - Check network connectivity from instance via floating IP (failure) Failed to create rule in security group. Please refer to OpenStack logs for more details.

In ostf log we can see that test failed in 00:51 by timeout:
 00:51:00 FAILURE Check network connectivity from instance via floating IP
File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib64/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 194, in _raise_TimeOut
    raise TimeOutError()
TimeOutError

At the same time, in nova log we can see that exactly in 00:51 rule was added, but we interupt the test and send delete request:

016-01-11T00:51:09.192093+00:00 debug: 2016-01-11 00:51:09.190 17855 DEBUG keystoneclient.session [req-80d18190-cb98-41d1-aeeb-f2ec9e698a72 0abb6a958ef64155992546a567b8665f 6d52f3fd0d314df88dfe3f69c2e33375 - - -] RESP: [200] date: Mon, 11 Jan 2016 00:51:09 GMT connection: close content-type: application/json; charset=UTF-8 content-length: 1224 x-openstack-request-id: req-9569a914-15f5-4625-b3d1-3a82fe922ee5
RESP BODY: {"security_group": {"tenant_id": "6d52f3fd0d314df88dfe3f69c2e33375", "description": "ost1_test-secgroup-smoke-netw1597447401 description", "id": "e72f8469-3475-4954-bc63-5065bb4da63c", "security_group_rules": [{"remote_group_id": null, "direction": "egress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv6", "tenant_id": "6d52f3fd0d314df88dfe3f69c2e33375", "port_range_max": null, "port_range_min": null, "id": "1c65e0bd-5076-40b6-8997-784f24183caa", "security_group_id": "e72f8469-3475-4954-bc63-5065bb4da63c"}, {"remote_group_id": null, "direction": "egress", "remote_ip_prefix": null, "protocol": null, "ethertype": "IPv4", "tenant_id": "6d52f3fd0d314df88dfe3f69c2e33375", "port_range_max": null, "port_range_min": null, "id": "c2c4069a-bb38-4de1-9cc6-21d0d31f1de5", "security_group_id": "e72f8469-3475-4954-bc63-5065bb4da63c"}, {"remote_group_id": null, "direction": "ingress", "remote_ip_prefix": "0.0.0.0/0", "protocol": "tcp", "ethertype": "IPv4", "tenant_id": "6d52f3fd0d314df88dfe3f69c2e33375", "port_range_max": 22, "port_range_min": 22, "id": "fa670538-f41f-48a3-8a9a-8176d7e67356", "security_group_id": "e72f8469-3475-4954-bc63-5065bb4da63c"}], "name": "ost1_test-secgroup-smoke-netw1597447401"}}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneclient/session.py:215
2016-01-11T00:51:09.194155+00:00 debug: 2016-01-11 00:51:09.192 17855 DEBUG keystoneclient.session [req-80d18190-cb98-41d1-aeeb-f2ec9e698a72 0abb6a958ef64155992546a567b8665f 6d52f3fd0d314df88dfe3f69c2e33375 - - -] REQ: curl -g -i -X DELETE http://10.109.11.3:9696/v2.0/security-groups/e72f8469-3475-4954-bc63-5065bb4da63c.json -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}83371d59a49e73a7f1f63646c917f5b2b03db2ae" _http_log_request /usr/lib/python2.7/dist-packages/keystoneclient/session.py:198
2016-01-11T00:51:09.324869+00:00 debug: 2016-01-11 00:51:09.320 17855 DEBUG keystoneclient.session [req-80d18190-cb98-41d1-aeeb-f2ec9e698a72 0abb6a958ef64155992546a567b8665f 6d52f3fd0d314df88dfe3f69c2e33375 - - -] RESP: [204] date: Mon, 11 Jan 2016 00:51:09 GMT connection: close content-length: 0 x-openstack-request-id: req-38c1ccad-943e-427c-a654-c2094939987c _http_log_response /usr/lib/python2.7/dist-packages/keystoneclient/session.py:215
2016-01-11T00:51:09.325368+00:00 info: 2016-01-11 00:51:09.323 17855 INFO nova.osapi_compute.wsgi.server [req-80d18190-cb98-41d1-aeeb-f2ec9e698a72 0abb6a958ef64155992546a567b8665f 6d52f3fd0d314df88dfe3f69c2e33375 - - -] 10.109.11.3 "DELETE /v2/6d52f3fd0d314df88dfe3f69c2e33375/os-security-groups/e72f8469-3475-4954-bc63-5065bb4da63c HTTP/1.1" status: 202 len: 196 time: 2.9439869

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "417"
  build_id: "417"
  fuel-nailgun_sha: "9ebbaa0473effafa5adee40270da96acf9c7d58a"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "df16d41cd7a9445cf82ad9fd8f0d53824711fcd8"
  fuel-nailgun-agent_sha: "92ebd5ade6fab60897761bfa084aefc320bff246"
  astute_sha: "c7ca63a49216744e0bfdfff5cb527556aad2e2a5"
  fuel-library_sha: "7ef751bdc0e4601310e85b8bf713a62ed4aee305"
  fuel-ostf_sha: "214e794835acc7aa0c1c5de936e93696a90bb57a"
  fuel-mirror_sha: "b62f3cce5321fd570c6589bc2684eab994c3f3f2"
  fuelmenu_sha: "2a0def56276f0fc30fd949605eeefc43e5d7cc6c"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "9f0ba4577915ce1e77f5dc9c639a5ef66ca45896"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "cfeadd34d8d048deeabf0884931708b1d040b8a6"

Tags: area-qa
Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Ilya Kutukov (ikutukov)
Changed in fuel:
status: New → Confirmed
Aleksei Stepanov (penguinolog)
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Alexey Stepanov (astepanov-m)
Aleksei Stepanov (penguinolog)
Changed in fuel:
status: Confirmed → Invalid
Revision history for this message
Aleksei Stepanov (penguinolog) wrote :

ISO fuel-8.0-574-2016-02-16_13-42-00.iso:
Probably main failure is generated by too fast OSTF start:
Failed tests only:
1. start :

 Create volume and boot instance from it failure
 Check state of haproxy backends on controllers failure
 Check that required services are running failure
 Check network connectivity from instance via floating IP failure

2 and more:
 Check that required services are running failure
 Check that required services are running failure

Seeking for safe way to pause, until cluster will return to the working state.

Revision history for this message
Aleksei Stepanov (penguinolog) wrote :

time.sleep(300) after 'wait_mysql_galera_is_up' is an extremely dirty hack, which fixes the test.

Changed in fuel:
status: Invalid → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/282287

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Alexandr Kostrikov (akostrikov-mirantis) wrote :

Reproduced:
https://mirantis.testrail.com/index.php?/tests/view/2894605

```
fuel_health.common.test_mixins: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuel_health/common/test_mixins.py", line 177, in verify
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 625, in _create_security_group
    self.fail("Failed to create rule in security group.")
  File "/usr/lib/python2.7/site-packages/unittest2/case.py", line 666, in fail
    raise self.failureException(msg)
AssertionError: Failed to create rule in security group.
```

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Download full text (4.1 KiB)

necessary part of trace
keystoneclient.auth.identity.v2: DEBUG: Making authentication request to https://10.109.3.3:5000/v2.0/tokens
fuel_health.nmanager: DEBUG: [<Image: TestVM>]
fuel_health.nmanager: DEBUG: TestVM
fuel_health.common.test_mixins: INFO: STEP:1, verify action: 'security group creation'
fuel_health.nmanager: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 622, in _create_security_group
    client.security_group_rules.create(secgroup.id, **ruleset)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/security_group_rules.py", line 70, in create
    'security_group_rule')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create
    _resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 449, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 424, in _cs_request
    resp, body = self._time_request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 397, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 366, in request
    **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1051, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 415, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 371, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 196, in recv
    read = self.read(buflen)
  File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 136, in read
    super(GreenSSLSocket, self).read, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 116, in _call_trampolining
    timeout_exc=timeout_exc('timed out'))
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 162, in trampoline
    return hub.switch()
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
  File "/usr/lib/python2.7/site-p...

Read more...

Revision history for this message
Tatyanka (tatyana-leontovich) wrote :
Download full text (4.1 KiB)

necessary part of trace
keystoneclient.auth.identity.v2: DEBUG: Making authentication request to https://10.109.3.3:5000/v2.0/tokens
fuel_health.nmanager: DEBUG: [<Image: TestVM>]
fuel_health.nmanager: DEBUG: TestVM
fuel_health.common.test_mixins: INFO: STEP:1, verify action: 'security group creation'
fuel_health.nmanager: DEBUG: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/fuel_health/nmanager.py", line 622, in _create_security_group
    client.security_group_rules.create(secgroup.id, **ruleset)
  File "/usr/lib/python2.7/site-packages/novaclient/v2/security_group_rules.py", line 70, in create
    'security_group_rule')
  File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create
    _resp, body = self.api.client.post(url, body=body)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 449, in post
    return self._cs_request(url, 'POST', **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 424, in _cs_request
    resp, body = self._time_request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 397, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 366, in request
    **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1051, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 415, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 371, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 196, in recv
    read = self.read(buflen)
  File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 136, in read
    super(GreenSSLSocket, self).read, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/eventlet/green/ssl.py", line 116, in _call_trampolining
    timeout_exc=timeout_exc('timed out'))
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 162, in trampoline
    return hub.switch()
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
  File "/usr/lib/python2.7/site-p...

Read more...

Revision history for this message
Aleksei Stepanov (penguinolog) wrote :

Floating bug, with difficult to find of marker, that services are really ready.

Revision history for this message
Alexey. Kalashnikov (akalashnikov) wrote :

Reproduced on swarm test:
9.1 snapshot #136
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.repetitive_restart/28/console

ostf log cut:
http://paste.openstack.org/show/557440/

diagnostic snapshot:
https://drive.google.com/a/mirantis.com/file/d/0B0EB6QSDWt2vVzMxZXFlTVFyRTQ/view?usp=sharing

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-qa (master)

Change abandoned by Alexey Stepanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/282287
Reason: abandon

Revision history for this message
Dmitry Belyaninov (dbelyaninov) wrote :

https://product-ci.infra.mirantis.net/job/9.x.acceptance.ubuntu.ha_scale_group_2/30/testReport/(root)/remove_controllers/remove_controllers/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.