[System tests] Test on Murano fails with ConnectionError: HTTPConnectionPool

[Andrey Sledzinskiy]

http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.services/45/testReport/junit/%28root%29/deploy_murano_simple/deploy_murano_simple/

Test on Murano fails with ConnectionError: HTTPConnectionPool(host='10.108.1.3', port=8774): Max retries exceeded with url: /v2/b5e0206531b2482ca1cc158d9c6326e3/os-security-groups (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

Test fails on common_func.goodbye_security() method call with trace

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/usr/lib/python2.7/dist-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/usr/lib/python2.7/dist-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/helpers/decorators.py", line 49, in wrapper
    return func(*args, **kwagrs)
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/tests/test_services.py", line 205, in deploy_murano_simple
    common_func.goodbye_security()
  File "/home/jenkins/workspace/master_fuelmain.system_test.centos.services/fuelweb_test/helpers/common.py", line 53, in goodbye_security
    secgroup = self.nova.security_groups.find(name='default')
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 190, in find
    matches = self.findall(**kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 228, in findall
    listing = self.list(**list_kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/security_groups.py", line 96, in list
    'security_groups')
  File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 64, in _list
    _resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 283, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
    resp, body = self.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 213, in request
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='10.108.1.3', port=8774): Max retries exceeded with url: /v2/b5e0206531b2482ca1cc158d9c6326e3/os-security-groups (Caused by <class 'socket.error'>: [Errno 104] Connection reset by peer)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/94190

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/94190
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=c2026435eb727df73ecf4a2a118c5a848abfc429
Submitter: Jenkins
Branch: master

commit c2026435eb727df73ecf4a2a118c5a848abfc429
Author: Tatyana Leontovich <email address hidden>
Date: Mon May 19 16:13:00 2014 +0000

    Add security group creation in service tests

    add creation of separate security group
    for service tests. Also Increase timeout for
    migration vm functionality and add more logging
    to improve debug of this tests

    Change-Id: Ia310f57a98c4d4aa11138f76cd9e0913accfbebc
    Closes-Bug: #1320181
    Related-Bug: #1317548

[Timur Nurlygayanov (tnurlygayanov)]

Andrey, can you please provide the environment with this problem?

Also, looks like this fix does not related to this problem.

[Andrey Sledzinskiy (asledzinskiy)]

Looks like this problem isn't reproduced in latest system test results and test fails with timeout on OSTF run

http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.services/56/testReport/junit/%28root%29/deploy_murano_simple/deploy_murano_simple/

[Andrey Sledzinskiy (asledzinskiy)]

Issue was reproduced in latest sys test run http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.services_simple/1/testReport/junit/%28root%29/deploy_murano_simple/deploy_murano_simple/

Logs are attached

[Timur Nurlygayanov (tnurlygayanov)]

We have inwestigated this issue and looks like the root of this problem not in Murano or Sahara.
What we can see right now: during the system tests Nova API returns the exception about too many requests.

Need to investigate how it is possible and why Nova can not return the list of security groups.

[Tatyanka (tatyana-leontovich)]

This problem happens only in system test for murano, so please, could you fix the test - or localizate problem in murano deployment (btw I am not sure whe we need permit all traffic using method goodbay security) if we just run ostf that should work fine without it. May be it make sense leave this method only for sahara test?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/94831

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/94831
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=703a1820b9fa3d3b99cf425921494a4b74f3fbbb
Submitter: Jenkins
Branch: master

commit 703a1820b9fa3d3b99cf425921494a4b74f3fbbb
Author: TimurNurlygayanov <email address hidden>
Date: Thu May 22 15:38:25 2014 +0400

    Fixed issue with extra function

    Fixed issue with goodbye_security function, which shouldn't be here.

    Change-Id: I76362607b2d879bf053bbece304cd0ac337b8008
    Closes-Bug: #1320181

[Andrey Sledzinskiy (asledzinskiy)]

Looks like issue wasn't fixed. Still get this error
http://jenkins-product.srt.mirantis.net:8080/view/5.0_swarm/job/5.0_fuelmain.system_test.centos.services_simple/4/testReport/junit/%28root%29/deploy_murano_simple/deploy_murano_simple/
Logs are attached

[Timur Nurlygayanov (tnurlygayanov)]

This is not murano issue.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/96733

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/96733
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=aa0d05053d044508b97b8ab5501aa950cbc5c75c
Submitter: Jenkins
Branch: master

commit aa0d05053d044508b97b8ab5501aa950cbc5c75c
Author: Tatyana Leontovich <email address hidden>
Date: Fri May 30 14:47:04 2014 +0000

    Change not existing tests classes in murano test

    We try to invoke non existing tests here,
    in part run ostf.
    Change murano ostf test class path.
    Also delete unused create keypair method

    Change-Id: Iadcc83a72f5cb9071be3838e12d03a6537cd213e
    Related-Bug: #1320181

[Tatyanka (tatyana-leontovich)]

fixed for 5.1 http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.ubuntu.services_simple/8/testReport/%28root%29/deploy_murano_simple/
should be added in stable/5.0

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/5.0
Review: https://review.openstack.org/97741

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/97741
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=c9996907a669d842bbfa46403592e2e898b3e65c
Submitter: Jenkins
Branch: stable/5.0

commit c9996907a669d842bbfa46403592e2e898b3e65c
Author: Tatyana Leontovich <email address hidden>
Date: Fri May 30 14:47:04 2014 +0000

    5.0.1 Change not existing tests classes in murano test

    We try to invoke non existing tests here,
    in part run ostf.
    Change murano ostf test class path.
    Also delete unused create keypair method

    Change-Id: Iadcc83a72f5cb9071be3838e12d03a6537cd213e
    Related-Bug: #1320181
    (cherry picked from commit aa0d05053d044508b97b8ab5501aa950cbc5c75c)

[Dmitry Borodaenko (angdraug)]

Why is priority inconsistent between release series? Is impact of this problem smaller in 4.1 and 5.0?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/98466

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/98468

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/98466
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=ae9b45c06e8c10d69b95749e69090172bf10d48e
Submitter: Jenkins
Branch: stable/4.1

commit ae9b45c06e8c10d69b95749e69090172bf10d48e
Author: TimurNurlygayanov <email address hidden>
Date: Thu May 22 15:38:25 2014 +0400

    Fixed issue with extra function

    Fixed issue with goodbye_security function, which shouldn't be here.

    Change-Id: I76362607b2d879bf053bbece304cd0ac337b8008
    Closes-Bug: #1320181
    (cherry picked from commit 703a1820b9fa3d3b99cf425921494a4b74f3fbbb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/98468
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=91b41af3f66128e90b994a135f570f585a17b241
Submitter: Jenkins
Branch: stable/4.1

commit 91b41af3f66128e90b994a135f570f585a17b241
Author: Tatyana Leontovich <email address hidden>
Date: Fri May 30 14:47:04 2014 +0000

    Change not existing tests classes in murano test

    We try to invoke non existing tests here,
    in part run ostf.
    Change murano ostf test class path.
    Also delete unused create keypair method

    Change-Id: Iadcc83a72f5cb9071be3838e12d03a6537cd213e
    Related-Bug: #1320181

[Tatyanka (tatyana-leontovich)]

Still reproduce here http://jenkins-product.srt.mirantis.net:8080/view/0_master_swarm/job/master_fuelmain.system_test.ubuntu.services_simple/66/testReport/junit/%28root%29/deploy_murano_simple/deploy_murano_simple/

[Artem Panchenko (apanchenko-8)]

Also reproduced for 5.0.x:

http://jenkins-product.srt.mirantis.net:8080/view/5.0_swarm/job/5.0_fuelmain.system_test.ubuntu.services_simple/66/testReport/(root)/deploy_murano_simple/deploy_murano_simple/

[Timur Nurlygayanov (tnurlygayanov)]

Looks like this issue no reproduced now for all tests, so, status changed to 'fix commited'.