MapRFS test__execute_local fails trying ssh connect

Bug #1659162 reported by Mark Sturdevant
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Critical
Valeriy Ponomaryov

Bug Description

The unit test does not appear to always follow the expected code path, so it sometimes fails.

==============================
2017-01-24 22:10:05.935041 | Failed 1 tests - output below:
2017-01-24 22:10:05.935052 | ==============================
2017-01-24 22:10:05.935057 |
2017-01-24 22:10:05.935090 | manila.tests.share.drivers.maprfs.test_maprfs.MapRFSNativeShareDriverTestCase.test__execute_local
2017-01-24 22:10:05.935115 | -------------------------------------------------------------------------------------------------
2017-01-24 22:10:05.935121 |
2017-01-24 22:10:05.935130 | Captured pythonlogging:
2017-01-24 22:10:05.935140 | ~~~~~~~~~~~~~~~~~~~~~~~
2017-01-24 22:10:05.935167 | b' ERROR [manila.utils] Check whether private key or password are correctly set. Error connecting via ssh: '
2017-01-24 22:10:05.935175 | b''
2017-01-24 22:10:05.935180 |
2017-01-24 22:10:05.935189 | Captured traceback:
2017-01-24 22:10:05.935198 | ~~~~~~~~~~~~~~~~~~~
2017-01-24 22:10:05.935211 | b'Traceback (most recent call last):'
2017-01-24 22:10:05.935235 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/utils.py", line 107, in create'
2017-01-24 22:10:05.935249 | b' timeout=self.conn_timeout)'
2017-01-24 22:10:05.935282 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/paramiko/client.py", line 305, in connect'
2017-01-24 22:10:05.935299 | b' retry_on_signal(lambda: sock.connect(addr))'
2017-01-24 22:10:05.935333 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/paramiko/util.py", line 269, in retry_on_signal'
2017-01-24 22:10:05.935344 | b' return function()'
2017-01-24 22:10:05.935377 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/paramiko/client.py", line 305, in <lambda>'
2017-01-24 22:10:05.935392 | b' retry_on_signal(lambda: sock.connect(addr))'
2017-01-24 22:10:05.935426 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/greenio/base.py", line 252, in connect'
2017-01-24 22:10:05.935441 | b' timeout_exc=socket.timeout("timed out"))'
2017-01-24 22:10:05.935475 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/greenio/base.py", line 203, in _trampoline'
2017-01-24 22:10:05.935490 | b' mark_as_closed=self._mark_as_closed)'
2017-01-24 22:10:05.935524 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/hubs/__init__.py", line 162, in trampoline'
2017-01-24 22:10:05.935535 | b' return hub.switch()'
2017-01-24 22:10:05.935568 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/hubs/hub.py", line 294, in switch'
2017-01-24 22:10:05.935581 | b' return self.greenlet.switch()'
2017-01-24 22:10:05.935613 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/hubs/hub.py", line 346, in run'
2017-01-24 22:10:05.935625 | b' self.wait(sleep_time)'
2017-01-24 22:10:05.935657 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/hubs/poll.py", line 85, in wait'
2017-01-24 22:10:05.935670 | b' presult = self.do_poll(seconds)'
2017-01-24 22:10:05.935704 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll'
2017-01-24 22:10:05.935717 | b' return self.poll.poll(seconds)'
2017-01-24 22:10:05.935753 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler'
2017-01-24 22:10:05.935766 | b' raise TimeoutException()'
2017-01-24 22:10:05.935781 | b'fixtures._fixtures.timeout.TimeoutException'
2017-01-24 22:10:05.935787 | b''
2017-01-24 22:10:05.935806 | b'During handling of the above exception, another exception occurred:'
2017-01-24 22:10:05.935813 | b''
2017-01-24 22:10:05.935826 | b'Traceback (most recent call last):'
2017-01-24 22:10:05.935856 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/share/drivers/maprfs/driver_util.py", line 65, in _execute'
2017-01-24 22:10:05.935877 | b' out, err = self._run_ssh(host, cmd, check_exit_code)'
2017-01-24 22:10:05.935907 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/share/drivers/maprfs/driver_util.py", line 104, in _run_ssh'
2017-01-24 22:10:05.935918 | b' max_size=max_size)'
2017-01-24 22:10:05.935943 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/utils.py", line 88, in __init__'
2017-01-24 22:10:05.935959 | b' super(SSHPool, self).__init__(*args, **kwargs)'
2017-01-24 22:10:05.935991 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/pools.py", line 77, in __init__'
2017-01-24 22:10:05.936006 | b' self.free_items.append(self.create())'
2017-01-24 22:10:05.936030 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/utils.py", line 124, in create'
2017-01-24 22:10:05.936044 | b' raise exception.SSHException(msg)'
2017-01-24 22:10:05.936072 | b'manila.exception.SSHException: Check whether private key or password are correctly set. Error connecting via ssh: '
2017-01-24 22:10:05.936079 | b''
2017-01-24 22:10:05.936098 | b'During handling of the above exception, another exception occurred:'
2017-01-24 22:10:05.936105 | b''
2017-01-24 22:10:05.936118 | b'Traceback (most recent call last):'
2017-01-24 22:10:05.936151 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/tests/share/drivers/maprfs/test_maprfs.py", line 631, in test__execute_local'
2017-01-24 22:10:05.936168 | b' self._driver._maprfs_util._execute("fake", "cmd")'
2017-01-24 22:10:05.936198 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/share/drivers/maprfs/driver_util.py", line 83, in _execute'
2017-01-24 22:10:05.936216 | b' raise exception.ProcessExecutionError(six.text_type(e))'
2017-01-24 22:10:05.936240 | b'oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.'
2017-01-24 22:10:05.936249 | b'Command: None'
2017-01-24 22:10:05.936257 | b'Exit code: -'
2017-01-24 22:10:05.936282 | b"Stdout: 'Check whether private key or password are correctly set. Error connecting via ssh: '"
2017-01-24 22:10:05.936291 | b'Stderr: None'
2017-01-24 22:10:05.936297 | b''
2017-01-24 22:10:05.936302 |

Changed in manila:
assignee: nobody → Mark Sturdevant (mark-sturdevant)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.openstack.org/424932

Changed in manila:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.openstack.org/424932
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=399d596d3029575210f5d5483326e78ac6abc71a
Submitter: Jenkins
Branch: master

commit 399d596d3029575210f5d5483326e78ac6abc71a
Author: Mark Sturdevant <email address hidden>
Date: Tue Jan 24 17:54:17 2017 -0800

    Fix MapRFS test__execute to not impact others

    MapRFS test__execute_local() can fail trying ssh connect
    when it is supposed to only do local tests.
    The host list can be modified by other tests. So,
    fix the test that modifies hosts. Make it use its own
    instance of utils so it won't affect others.

    Add a little more validation too.

    Change-Id: I42bc2247c94490c9a0aad6c401b0fe44ccd99030
    Closes-Bug: #1659162

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 4.0.0.0b3

This issue was fixed in the openstack/manila 4.0.0.0b3 development milestone.

Revision history for this message
Valeriy Ponomaryov (vponomaryov) wrote :
Changed in manila:
status: Fix Released → New
milestone: none → pike-1
importance: Undecided → Critical
Changed in manila:
assignee: Mark Sturdevant (mark-sturdevant) → Valeriy Ponomaryov (vponomaryov)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.openstack.org/440531
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=e79cf7bf3302aadc0031806c964bab077f64994b
Submitter: Jenkins
Branch: master

commit e79cf7bf3302aadc0031806c964bab077f64994b
Author: vponomaryov <email address hidden>
Date: Thu Mar 2 14:53:59 2017 +0200

    Fix 3 CI breakages

    1) Increase default RAM value in manila flavor to be able
       to use manila service image with new 64bit architecture.
    2) Fix test requirement on "hacking" lib to avoid pep8 breakage.
    3) Fix unstable maprfs driver unit test that depends on local hostnames
       and IPs.

    Change-Id: Ib36113e33e3616984be5864fbaf73c1fc5d7a315
    Closes-Bug: #1669412
    Closes-Bug: #1659162

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/441253

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/ocata)

Reviewed: https://review.openstack.org/441253
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=5d31673094327ff8c55568a3279790e87ef6d003
Submitter: Jenkins
Branch: stable/ocata

commit 5d31673094327ff8c55568a3279790e87ef6d003
Author: vponomaryov <email address hidden>
Date: Thu Mar 2 14:53:59 2017 +0200

    Fix 3 CI breakages

    1) Increase default RAM value in manila flavor to be able
       to use manila service image with new 64bit architecture.
    2) Fix test requirement on "hacking" lib to avoid pep8 breakage.
    3) Fix unstable maprfs driver unit test that depends on local hostnames
       and IPs.

    Change-Id: Ib36113e33e3616984be5864fbaf73c1fc5d7a315
    Closes-Bug: #1669412
    Closes-Bug: #1659162
    (cherry picked from commit e79cf7bf3302aadc0031806c964bab077f64994b)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 5.0.0.0b1

This issue was fixed in the openstack/manila 5.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 4.0.1

This issue was fixed in the openstack/manila 4.0.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.