The remote interface (savanna/utils/remote.py) opens and closes the SSH/SFTP connection per interaction (e.g. execute_command()). In our testing, this constant re-opening and closing of connections yields the following error:
2013-08-13 23:46:40.288 6711 ERROR paramiko.transport [-]
Traceback (most recent call last):
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 346, in fire_timers
timer()
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/eventlet/hubs/timer.py", line 56, in __call__
cb(*args, **kw)
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/eventlet/semaphore.py", line 121, in _do_acquire
waiter.switch()
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
result = function(*args, **kwargs)
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/savanna/context.py", line 132, in wrapper
func(*args, **kwargs)
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/savanna/plugins/hdp/hadoopserver.py", line 37, in provision_ambari
self._setup_and_start_ambari_agent(ambari_info.host.internal_ip)
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/savanna/plugins/hdp/hadoopserver.py", line 90, in _setup_and_start_ambari_agent
ambari_server_ip)
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/savanna/utils/remote.py", line 131, in replace_remote_string
with contextlib.closing(self.ssh_connection()) as ssh:
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/savanna/utils/remote.py", line 112, in ssh_connection
self.instance.node_group.cluster.private_key)
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/savanna/utils/remote.py", line 31, in setup_ssh_connection
ssh.connect(host, username=username, pkey=private_key)
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/paramiko/client.py", line 311, in connect
t.start_client()
File "/root/dev/savanna/.tox/venv/lib/python2.6/site-packages/paramiko/transport.py", line 465, in start_client
raise e
SSHException: Error reading SSH protocol bannerSecond simultaneous read on fileno 9 detected. Unless you really know what you're doing, make sure that only one greenthread can read any particular socket. Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_prevent_multiple_readers(False)
Although the error seems to indicate a threading issue, we have clearly seen from our testing that the actual cause is related to the frequent opening and closing connections.
Rather than opening and closing these connections on a per invocation basis, the code should be modified to leverage a single SSH/SFTP connection for the duration of its use.
The issue needs further digging, but I can suggest a workaround for now:
Change ambariplugin.py in the following way:
def _spawn(self, description, func, args):
func(args)
That will make deployment single-threaded