[System tests] Test on Murano fails with ConnectionError: HTTPConnectionPool

Bug #1320181 reported by Andrey Sledzinskiy on 2014-05-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
High
Fuel QA Team
4.1.x
Medium
Fuel QA Team
5.0.x
Medium
Fuel QA Team

Bug Description

http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.services/45/testReport/junit/%28root%29/deploy_murano_simple/deploy_murano_simple/

Test on Murano fails with ConnectionError: HTTPConnectionPool(host='10.108.1.3', port=8774): Max retries exceeded with url: /v2/b5e0206531b2482ca1cc158d9c6326e3/os-security-groups (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

Test fails on common_func.goodbye_security() method call with trace

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/helpers/decorators.py", line 49, in wrapper
    return func(*args, **kwagrs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/tests/test_services.py", line 205, in deploy_murano_simple
    common_func.goodbye_security()
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/helpers/common.py", line 53, in goodbye_security
    secgroup = self.nova.security_groups.find(name='default')
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 190, in find
    matches = self.findall(**kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 228, in findall
    listing = self.list(**list_kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/security_groups.py", line 96, in list
    'security_groups')
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 283, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 213, in request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='10.108.1.3', port=8774): Max retries exceeded with url: /v2/b5e0206531b2482ca1cc158d9c6326e3/os-security-groups (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

Mike Scherbakov (mihgen) on 2014-05-16
summary: [System tests] Test on Murano fails with ConnectionError:
- HTTPConnectionPool(host='10.108.1.3', port=8774): Max retries exceeded
- with url: /v2/b5e0206531b2482ca1cc158d9c6326e3/os-security-groups
- (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)
+ HTTPConnectionPool
description: updated
Mike Scherbakov (mihgen) on 2014-05-16
Changed in fuel:
milestone: none → 5.0
milestone: 5.0 → 5.1
Changed in fuel:
status: New → Confirmed
assignee: Fuel QA Team (fuel-qa) → Tatyana (tatyana-leontovich)

Fix proposed to branch: master
Review: https://review.openstack.org/94190

Changed in fuel:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/94190
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=c2026435eb727df73ecf4a2a118c5a848abfc429
Submitter: Jenkins
Branch: master

commit c2026435eb727df73ecf4a2a118c5a848abfc429
Author: Tatyana Leontovich <email address hidden>
Date: Mon May 19 16:13:00 2014 +0000

    Add security group creation in service tests

    add creation of separate security group
    for service tests. Also Increase timeout for
    migration vm functionality and add more logging
    to improve debug of this tests

    Change-Id: Ia310f57a98c4d4aa11138f76cd9e0913accfbebc
    Closes-Bug: #1320181
    Related-Bug: #1317548

Changed in fuel:
status: In Progress → Fix Committed
Changed in fuel:
status: Fix Committed → Confirmed
Changed in fuel:
assignee: Tatyana (tatyana-leontovich) → Timur Nurlygayanov (tnurlygayanov)

Andrey, can you please provide the environment with this problem?

Also, looks like this fix does not related to this problem.

Changed in fuel:
assignee: Timur Nurlygayanov (tnurlygayanov) → Andrey Sledzinskiy (asledzinskiy)
status: Confirmed → Incomplete
Changed in fuel:
assignee: Andrey Sledzinskiy (asledzinskiy) → Timur Nurlygayanov (tnurlygayanov)
status: Incomplete → Confirmed
Changed in fuel:
assignee: Timur Nurlygayanov (tnurlygayanov) → Fuel OSTF Team (fuel-ostf)
Changed in fuel:
assignee: Fuel OSTF Team (fuel-ostf) → Timur Nurlygayanov (tnurlygayanov)

We have inwestigated this issue and looks like the root of this problem not in Murano or Sahara.
What we can see right now: during the system tests Nova API returns the exception about too many requests.

Need to investigate how it is possible and why Nova can not return the list of security groups.

Tatyanka (tatyana-leontovich) wrote :

This problem happens only in system test for murano, so please, could you fix the test - or localizate problem in murano deployment (btw I am not sure whe we need permit all traffic using method goodbay security) if we just run ostf that should work fine without it. May be it make sense leave this method only for sahara test?

Fix proposed to branch: master
Review: https://review.openstack.org/94831

Changed in fuel:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/94831
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=703a1820b9fa3d3b99cf425921494a4b74f3fbbb
Submitter: Jenkins
Branch: master

commit 703a1820b9fa3d3b99cf425921494a4b74f3fbbb
Author: TimurNurlygayanov <email address hidden>
Date: Thu May 22 15:38:25 2014 +0400

    Fixed issue with extra function

    Fixed issue with goodbye_security function, which shouldn't be here.

    Change-Id: I76362607b2d879bf053bbece304cd0ac337b8008
    Closes-Bug: #1320181

Changed in fuel:
status: In Progress → Fix Committed
Changed in fuel:
status: Fix Committed → Confirmed
tags: added: murano

This is not murano issue.

Reviewed: https://review.openstack.org/96733
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=aa0d05053d044508b97b8ab5501aa950cbc5c75c
Submitter: Jenkins
Branch: master

commit aa0d05053d044508b97b8ab5501aa950cbc5c75c
Author: Tatyana Leontovich <email address hidden>
Date: Fri May 30 14:47:04 2014 +0000

    Change not existing tests classes in murano test

    We try to invoke non existing tests here,
    in part run ostf.
    Change murano ostf test class path.
    Also delete unused create keypair method

    Change-Id: Iadcc83a72f5cb9071be3838e12d03a6537cd213e
    Related-Bug: #1320181

Tatyanka (tatyana-leontovich) wrote :
Changed in fuel:
assignee: Timur Nurlygayanov (tnurlygayanov) → Tatyana (tatyana-leontovich)
status: Confirmed → Fix Committed

Reviewed: https://review.openstack.org/97741
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=c9996907a669d842bbfa46403592e2e898b3e65c
Submitter: Jenkins
Branch: stable/5.0

commit c9996907a669d842bbfa46403592e2e898b3e65c
Author: Tatyana Leontovich <email address hidden>
Date: Fri May 30 14:47:04 2014 +0000

    5.0.1 Change not existing tests classes in murano test

    We try to invoke non existing tests here,
    in part run ostf.
    Change murano ostf test class path.
    Also delete unused create keypair method

    Change-Id: Iadcc83a72f5cb9071be3838e12d03a6537cd213e
    Related-Bug: #1320181
    (cherry picked from commit aa0d05053d044508b97b8ab5501aa950cbc5c75c)

Dmitry Borodaenko (angdraug) wrote :

Why is priority inconsistent between release series? Is impact of this problem smaller in 4.1 and 5.0?

Reviewed: https://review.openstack.org/98466
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=ae9b45c06e8c10d69b95749e69090172bf10d48e
Submitter: Jenkins
Branch: stable/4.1

commit ae9b45c06e8c10d69b95749e69090172bf10d48e
Author: TimurNurlygayanov <email address hidden>
Date: Thu May 22 15:38:25 2014 +0400

    Fixed issue with extra function

    Fixed issue with goodbye_security function, which shouldn't be here.

    Change-Id: I76362607b2d879bf053bbece304cd0ac337b8008
    Closes-Bug: #1320181
    (cherry picked from commit 703a1820b9fa3d3b99cf425921494a4b74f3fbbb)

Reviewed: https://review.openstack.org/98468
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=91b41af3f66128e90b994a135f570f585a17b241
Submitter: Jenkins
Branch: stable/4.1

commit 91b41af3f66128e90b994a135f570f585a17b241
Author: Tatyana Leontovich <email address hidden>
Date: Fri May 30 14:47:04 2014 +0000

    Change not existing tests classes in murano test

    We try to invoke non existing tests here,
    in part run ostf.
    Change murano ostf test class path.
    Also delete unused create keypair method

    Change-Id: Iadcc83a72f5cb9071be3838e12d03a6537cd213e
    Related-Bug: #1320181

Changed in fuel:
assignee: Tatyana (tatyana-leontovich) → nobody
assignee: nobody → Fuel QA Team (fuel-qa)

Looks like this issue no reproduced now for all tests, so, status changed to 'fix commited'.

Changed in fuel:
status: Confirmed → Fix Committed
Changed in fuel:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers