[devops][2.9] BadHostKeyException when time-sync because of paramiko update

Bug #1697690 reported by Vladimir Jigulin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Dennis Dmitriev

Bug Description

Used devops version: git+git://github.com/openstack/fuel-devops.git@2.9.23

Detailed bug description:
Changes introduced in paramiko==2.2.0
https://github.com/paramiko/paramiko/commit/4e973d97b0baf8fc9b7b5fbd84af4a869829dce5#diff-169e8b6af6ef7bee0bb2c9a6a637aa5f
cause time-sync fail when trying connect to slave node:

    self.env.revert_snapshot(snapshot_name)
  File "/home/jenkins/workspace/9.x.tempest.create_env/fuelweb_test/models/environment.py", line 321, in revert_snapshot
    self.sync_time()
  File "/home/jenkins/workspace/9.x.tempest.create_env/fuelweb_test/models/environment.py", line 146, in sync_time
    new_time = sync_time(self.d_env, nodes_names, skip_sync)
  File "/home/jenkins/qa-venv-9.2/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/qa-venv-9.2/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 36, in sync_time
    with GroupNtpSync(env, node_names) as g_ntp:
  File "/home/jenkins/qa-venv-9.2/local/lib/python2.7/site-packages/devops/helpers/ntp.py", line 293, in __init__
    remote = get_node_remote(env, node_name)
  File "/home/jenkins/qa-venv-9.2/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 185, in get_node_remote
    keys=get_private_keys(env)))
  File "/home/jenkins/qa-venv-9.2/local/lib/python2.7/site-packages/devops/helpers/ssh_client.py", line 251, in __call__
    auth=auth)
  File "/home/jenkins/qa-venv-9.2/local/lib/python2.7/site-packages/devops/helpers/ssh_client.py", line 365, in __init__
    self.__connect()
  File "/home/jenkins/qa-venv-9.2/local/lib/python2.7/site-packages/devops/helpers/retry.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/qa-venv-9.2/local/lib/python2.7/site-packages/devops/helpers/ssh_client.py", line 455, in __connect
    log=True)
  File "/home/jenkins/qa-venv-9.2/local/lib/python2.7/site-packages/devops/helpers/ssh_client.py", line 123, in connect
    client.connect(**kwargs)
  File "/home/jenkins/qa-venv-9.2/local/lib/python2.7/site-packages/paramiko/client.py", line 380, in connect
    raise BadHostKeyException(hostname, server_key, our_key)
BadHostKeyException: (u'10.109.13.3', <paramiko.ed25519key.Ed25519Key object at 0x7f0284af1810>, <paramiko.ed25519key.Ed25519Key object at 0x7f0284abeed0>)

Steps to reproduce:
dos.py time-sync envname
dos.py revert-resume envname snapshotname also affected
slaves (not admin) with installed system must be exists in env

Reproducibility: 100%
Workaround: Use paramiko==2.1.2

summary: - [devops] BadHostKeyException when time-sync because of paramiko update
+ [devops][2.9] BadHostKeyException when time-sync because of paramiko
+ update
Changed in fuel:
assignee: nobody → Fuel DevOps (fuel-devops)
importance: Undecided → Critical
Changed in fuel:
milestone: none → 9.x-updates
tags: added: blocker-for-qa
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-devops (release/2.9)

Fix proposed to branch: release/2.9
Review: https://review.openstack.org/474182

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-devops (master)

Fix proposed to branch: master
Review: https://review.openstack.org/474185

Changed in fuel:
assignee: Fuel DevOps (fuel-devops) → Dennis Dmitriev (ddmitriev)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-devops (release/2.9)

Reviewed: https://review.openstack.org/474182
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=d9facdcd975f6e2557a1bfaacdfff853c5d9e7e4
Submitter: Jenkins
Branch: release/2.9

commit d9facdcd975f6e2557a1bfaacdfff853c5d9e7e4
Author: Vladimir Khlyunev <email address hidden>
Date: Wed Jun 14 16:10:08 2017 +0400

    Ignore BadHostKeyException while trying ssh keys

    New version of paramiko uses BadHostKeyException as error when smth wrong
    with keys from known_hosts and current public key.

    Change-Id: I3ea9a1953b4e6daf33265600681c85cd2cd28828
    Closes-bug:1697690

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-devops (master)

Reviewed: https://review.openstack.org/474185
Committed: https://git.openstack.org/cgit/openstack/fuel-devops/commit/?id=3cb41214b566b116c4d33e906f3b3f25117a6a9b
Submitter: Jenkins
Branch: master

commit 3cb41214b566b116c4d33e906f3b3f25117a6a9b
Author: Vladimir Khlyunev <email address hidden>
Date: Wed Jun 14 16:10:08 2017 +0400

    Ignore BadHostKeyException while trying ssh keys

    New version of paramiko uses BadHostKeyException as error when smth wrong
    with keys from known_hosts and current public key.

    Change-Id: I3ea9a1953b4e6daf33265600681c85cd2cd28828
    Closes-bug:1697690

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to fuel-qa (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/474194

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-qa (stable/mitaka)

Reviewed: https://review.openstack.org/474194
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=1acd2425e2eeb10c26bd655e192b7d11cd22b36f
Submitter: Jenkins
Branch: stable/mitaka

commit 1acd2425e2eeb10c26bd655e192b7d11cd22b36f
Author: Vladimir Khlyunev <email address hidden>
Date: Wed Jun 14 17:05:13 2017 +0400

    Unlock fuel-devops' release/2.9 branch

    We can not tag commits anymore but we still need to
    fix bugs in fuel-devops from times to times and receive this fixes.

    Change-Id: I275a97343b0abe9d974f45fd313a44c0c42a862e
    Related-bug: 1697690

tags: added: in-stable-mitaka
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.