MapRFS test__execute_local fails trying ssh connect

[Mark Sturdevant]

The unit test does not appear to always follow the expected code path, so it sometimes fails.

==============================
2017-01-24 22:10:05.935041 | Failed 1 tests - output below:
2017-01-24 22:10:05.935052 | ==============================
2017-01-24 22:10:05.935057 |
2017-01-24 22:10:05.935090 | manila.tests.share.drivers.maprfs.test_maprfs.MapRFSNativeShareDriverTestCase.test__execute_local
2017-01-24 22:10:05.935115 | -------------------------------------------------------------------------------------------------
2017-01-24 22:10:05.935121 |
2017-01-24 22:10:05.935130 | Captured pythonlogging:
2017-01-24 22:10:05.935140 | ~~~~~~~~~~~~~~~~~~~~~~~
2017-01-24 22:10:05.935167 | b' ERROR [manila.utils] Check whether private key or password are correctly set. Error connecting via ssh: '
2017-01-24 22:10:05.935175 | b''
2017-01-24 22:10:05.935180 |
2017-01-24 22:10:05.935189 | Captured traceback:
2017-01-24 22:10:05.935198 | ~~~~~~~~~~~~~~~~~~~
2017-01-24 22:10:05.935211 | b'Traceback (most recent call last):'
2017-01-24 22:10:05.935235 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/utils.py", line 107, in create'
2017-01-24 22:10:05.935249 | b' timeout=self.conn_timeout)'
2017-01-24 22:10:05.935282 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/paramiko/client.py", line 305, in connect'
2017-01-24 22:10:05.935299 | b' retry_on_signal(lambda: sock.connect(addr))'
2017-01-24 22:10:05.935333 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/paramiko/util.py", line 269, in retry_on_signal'
2017-01-24 22:10:05.935344 | b' return function()'
2017-01-24 22:10:05.935377 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/paramiko/client.py", line 305, in <lambda>'
2017-01-24 22:10:05.935392 | b' retry_on_signal(lambda: sock.connect(addr))'
2017-01-24 22:10:05.935426 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/greenio/base.py", line 252, in connect'
2017-01-24 22:10:05.935441 | b' timeout_exc=socket.timeout("timed out"))'
2017-01-24 22:10:05.935475 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/greenio/base.py", line 203, in _trampoline'
2017-01-24 22:10:05.935490 | b' mark_as_closed=self._mark_as_closed)'
2017-01-24 22:10:05.935524 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/hubs/__init__.py", line 162, in trampoline'
2017-01-24 22:10:05.935535 | b' return hub.switch()'
2017-01-24 22:10:05.935568 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/hubs/hub.py", line 294, in switch'
2017-01-24 22:10:05.935581 | b' return self.greenlet.switch()'
2017-01-24 22:10:05.935613 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/hubs/hub.py", line 346, in run'
2017-01-24 22:10:05.935625 | b' self.wait(sleep_time)'
2017-01-24 22:10:05.935657 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/hubs/poll.py", line 85, in wait'
2017-01-24 22:10:05.935670 | b' presult = self.do_poll(seconds)'
2017-01-24 22:10:05.935704 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll'
2017-01-24 22:10:05.935717 | b' return self.poll.poll(seconds)'
2017-01-24 22:10:05.935753 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler'
2017-01-24 22:10:05.935766 | b' raise TimeoutException()'
2017-01-24 22:10:05.935781 | b'fixtures._fixtures.timeout.TimeoutException'
2017-01-24 22:10:05.935787 | b''
2017-01-24 22:10:05.935806 | b'During handling of the above exception, another exception occurred:'
2017-01-24 22:10:05.935813 | b''
2017-01-24 22:10:05.935826 | b'Traceback (most recent call last):'
2017-01-24 22:10:05.935856 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/share/drivers/maprfs/driver_util.py", line 65, in _execute'
2017-01-24 22:10:05.935877 | b' out, err = self._run_ssh(host, cmd, check_exit_code)'
2017-01-24 22:10:05.935907 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/share/drivers/maprfs/driver_util.py", line 104, in _run_ssh'
2017-01-24 22:10:05.935918 | b' max_size=max_size)'
2017-01-24 22:10:05.935943 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/utils.py", line 88, in __init__'
2017-01-24 22:10:05.935959 | b' super(SSHPool, self).__init__(*args, **kwargs)'
2017-01-24 22:10:05.935991 | b' File "/home/jenkins/workspace/gate-manila-python35-db/.tox/py35/lib/python3.5/site-packages/eventlet/pools.py", line 77, in __init__'
2017-01-24 22:10:05.936006 | b' self.free_items.append(self.create())'
2017-01-24 22:10:05.936030 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/utils.py", line 124, in create'
2017-01-24 22:10:05.936044 | b' raise exception.SSHException(msg)'
2017-01-24 22:10:05.936072 | b'manila.exception.SSHException: Check whether private key or password are correctly set. Error connecting via ssh: '
2017-01-24 22:10:05.936079 | b''
2017-01-24 22:10:05.936098 | b'During handling of the above exception, another exception occurred:'
2017-01-24 22:10:05.936105 | b''
2017-01-24 22:10:05.936118 | b'Traceback (most recent call last):'
2017-01-24 22:10:05.936151 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/tests/share/drivers/maprfs/test_maprfs.py", line 631, in test__execute_local'
2017-01-24 22:10:05.936168 | b' self._driver._maprfs_util._execute("fake", "cmd")'
2017-01-24 22:10:05.936198 | b' File "/home/jenkins/workspace/gate-manila-python35-db/manila/share/drivers/maprfs/driver_util.py", line 83, in _execute'
2017-01-24 22:10:05.936216 | b' raise exception.ProcessExecutionError(six.text_type(e))'
2017-01-24 22:10:05.936240 | b'oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.'
2017-01-24 22:10:05.936249 | b'Command: None'
2017-01-24 22:10:05.936257 | b'Exit code: -'
2017-01-24 22:10:05.936282 | b"Stdout: 'Check whether private key or password are correctly set. Error connecting via ssh: '"
2017-01-24 22:10:05.936291 | b'Stderr: None'
2017-01-24 22:10:05.936297 | b''
2017-01-24 22:10:05.936302 |

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/424932

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/424932
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=399d596d3029575210f5d5483326e78ac6abc71a
Submitter: Jenkins
Branch: master

commit 399d596d3029575210f5d5483326e78ac6abc71a
Author: Mark Sturdevant <email address hidden>
Date: Tue Jan 24 17:54:17 2017 -0800

    Fix MapRFS test__execute to not impact others

    MapRFS test__execute_local() can fail trying ssh connect
    when it is supposed to only do local tests.
    The host list can be modified by other tests. So,
    fix the test that modifies hosts. Make it use its own
    instance of utils so it won't affect others.

    Add a little more validation too.

    Change-Id: I42bc2247c94490c9a0aad6c401b0fe44ccd99030
    Closes-Bug: #1659162

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 4.0.0.0b3 development milestone.

[Valeriy Ponomaryov (vponomaryov)]

Observed again: http://logs.openstack.org/31/440531/1/check/gate-manila-python35/3e585db/console.html#_2017-03-02_13_15_40_690144

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/440531
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=e79cf7bf3302aadc0031806c964bab077f64994b
Submitter: Jenkins
Branch: master

commit e79cf7bf3302aadc0031806c964bab077f64994b
Author: vponomaryov <email address hidden>
Date: Thu Mar 2 14:53:59 2017 +0200

    Fix 3 CI breakages

    1) Increase default RAM value in manila flavor to be able
       to use manila service image with new 64bit architecture.
    2) Fix test requirement on "hacking" lib to avoid pep8 breakage.
    3) Fix unstable maprfs driver unit test that depends on local hostnames
       and IPs.

    Change-Id: Ib36113e33e3616984be5864fbaf73c1fc5d7a315
    Closes-Bug: #1669412
    Closes-Bug: #1659162

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/441253

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/441253
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=5d31673094327ff8c55568a3279790e87ef6d003
Submitter: Jenkins
Branch: stable/ocata

commit 5d31673094327ff8c55568a3279790e87ef6d003
Author: vponomaryov <email address hidden>
Date: Thu Mar 2 14:53:59 2017 +0200

    Fix 3 CI breakages

    1) Increase default RAM value in manila flavor to be able
       to use manila service image with new 64bit architecture.
    2) Fix test requirement on "hacking" lib to avoid pep8 breakage.
    3) Fix unstable maprfs driver unit test that depends on local hostnames
       and IPs.

    Change-Id: Ib36113e33e3616984be5864fbaf73c1fc5d7a315
    Closes-Bug: #1669412
    Closes-Bug: #1659162
    (cherry picked from commit e79cf7bf3302aadc0031806c964bab077f64994b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 5.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 4.0.1 release.