juju ssh/scp/run commands cause spurious key errors

Bug #802117 reported by Clint Byrum
66
This bug affects 12 people
Affects Status Importance Assigned to Milestone
Amulet
High
Unassigned
juju
Low
Unassigned
juju-deployer
Undecided
Unassigned
pyjuju
Low
Unassigned
juju (Ubuntu)
Medium
Unassigned

Bug Description

With lots of instances, one occasionally finds themselves using the same host name twice.

juju ssh should use a unique known_hosts file per environment, so as not to pollute the user's main known_hosts file.

Revision history for this message
Kapil Thangavelu (hazmat) wrote : Re: [Bug 802117] [NEW] ensemble ssh command should use a different known_hosts file

Excerpts from Clint Byrum's message of Sun Jun 26 08:51:09 UTC 2011:
> Public bug reported:
>
> With lots of instances, one occasionally finds themselves using the same
> twice.
>
> Ensemble ssh should use a unique known_hosts file per environment, so as
> not to pollute the user's main known_hosts file.
>
> ** Affects: ensemble
> Importance: Undecided
> Status: New
>

ideally ensemble should also seed and store the ssh host fingerprints to prevent the (imo) spurious ack question from ssh, likely in ~/.ensemble/ssh_hosts

Changed in ensemble:
importance: Undecided → Medium
Jim Baker (jimbaker)
Changed in juju:
milestone: none → florence
William Reade (fwereade)
Changed in juju:
status: New → In Progress
assignee: nobody → William Reade (fwereade)
Revision history for this message
Dustin Kirkland  (kirkland) wrote : Re: ensemble ssh command should use a different known_hosts file

I have some working shell code that addresses at least part of this, which you can tap for ideas. It generates SSH keys, adds fingerprints to a separate known hosts file, and prunes them when done with the instance.

It's not mergeable into Juju as is, but the functionality is quite nice. It's in a shell script called 'cloud-sandbox' in lp:bikeshed.

William Reade (fwereade)
Changed in juju:
status: In Progress → Confirmed
assignee: William Reade (fwereade) → nobody
Jim Baker (jimbaker)
Changed in juju:
status: Confirmed → In Progress
Changed in juju:
assignee: nobody → Jim Baker (jimbaker)
Changed in juju:
milestone: florence → galapagos
summary: - ensemble ssh command should use a different known_hosts file
+ juju ssh command should use a different known_hosts file
description: updated
summary: - juju ssh command should use a different known_hosts file
+ juju ssh/scp commands cause spurious key errors, should use a different
+ known_hosts file
Changed in juju (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Changed in juju:
milestone: galapagos → honolulu
Changed in juju:
assignee: Jim Baker (jimbaker) → nobody
status: In Progress → Confirmed
Changed in juju:
milestone: 0.6 → none
Martin Packman (gz)
Changed in juju-core:
importance: Undecided → Medium
status: New → Confirmed
Revision history for this message
Dave Cheney (dave-cheney) wrote : Re: juju ssh/scp commands cause spurious key errors, should use a different known_hosts file

Yes, we should do this.

Changed in juju-core:
assignee: nobody → Dave Cheney (dave-cheney)
milestone: none → 1.9.4
Changed in juju-core:
milestone: 1.9.4 → 1.9.5
Changed in juju-core:
milestone: 1.9.5 → 1.9.6
Changed in juju-core:
assignee: Dave Cheney (dave-cheney) → nobody
milestone: 1.9.6 → 1.9.7
status: Confirmed → Triaged
Changed in juju-core:
milestone: 1.9.7 → none
description: updated
Curtis Hovey (sinzui)
Changed in juju-core:
importance: Medium → Low
Curtis Hovey (sinzui)
Changed in juju:
status: Confirmed → Triaged
Curtis Hovey (sinzui)
tags: added: ssh
summary: - juju ssh/scp commands cause spurious key errors, should use a different
- known_hosts file
+ juju ssh/scp commands cause spurious key errors
Changed in juju:
importance: Medium → Low
Revision history for this message
Stuart Bishop (stub) wrote : Re: juju ssh/scp commands cause spurious key errors

A less obvious problem is that an invalid host key causes SSH tunneling to be disabled.

Revision history for this message
Stuart Bishop (stub) wrote :

Amulet tests are spuriously failing due to this bug. IP addresses get recycled as the leases expire, and the 'juju run' commands Amulet makes fail due to the old host key being in the root users known_hosts file.

======================================================================
ERROR: test suite for <class 'tests.test_integration.Test3UnitDeployment'>
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/stub/charms/cassandra/spike/.venv3/lib/python3.4/site-packages/nose/suite.py", line 210, in run
    self.setUp()
  File "/home/stub/charms/cassandra/spike/.venv3/lib/python3.4/site-packages/nose/suite.py", line 293, in setUp
    self.setupContext(ancestor)
  File "/home/stub/charms/cassandra/spike/.venv3/lib/python3.4/site-packages/nose/suite.py", line 316, in setupContext
    try_run(context, names)
  File "/home/stub/charms/cassandra/spike/.venv3/lib/python3.4/site-packages/nose/util.py", line 470, in try_run
    return func()
  File "/home/stub/charms/cassandra/spike/tests/test_integration.py", line 79, in setUpClass
    deployment.deploy(timeout=WAIT_TIMEOUT)
  File "/home/stub/charms/cassandra/spike/testing/amuletfixture.py", line 74, in deploy
    self.sentry.wait(timeout=timeout)
  File "/usr/lib/python3/dist-packages/amulet/sentry.py", line 259, in wait
    status = self.unit[unit].juju_agent()
  File "/usr/lib/python3/dist-packages/amulet/sentry.py", line 117, in juju_agent
    return self._run_unit_script("juju_agent.py")
  File "/usr/lib/python3/dist-packages/amulet/sentry.py", line 114, in _run_unit_script
    raise IOError(output)
OSError: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
04:1f:98:dc:d5:aa:3a:4d:aa:d8:f6:a2:15:e5:fa:29.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /root/.ssh/known_hosts:3
  remove with: ssh-keygen -f "/root/.ssh/known_hosts" -R 10.0.3.223
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
ERROR command timed out

Marco Ceppi (marcoceppi)
Changed in amulet:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Ryan Beisner (1chb1n) wrote :

FWIW - We saw this too in our automated OpenStack charm testing (UOSCI). Our work around is to overwrite known_hosts with our base known_hosts file on every build, on every jenkins slave. A bit of a hack, but it does the trick.

Marco Ceppi (marcoceppi)
tags: added: charmers
Marco Ceppi (marcoceppi)
Changed in amulet:
milestone: none → 1.10.0
Marco Ceppi (marcoceppi)
Changed in amulet:
milestone: 1.10.0 → 1.11.0
Curtis Hovey (sinzui)
summary: - juju ssh/scp commands cause spurious key errors
+ juju ssh/scp/run commands cause spurious key errors
Revision history for this message
Stuart Bishop (stub) wrote :

The root users .ssh/known_hosts is also getting polluted by 'juju run', which is causing subsequent 'juju run' commands to fail when IP addresses are recycled.

I'm also seeing IP addresses recycled much more often, especially with the local provider which now reuses them immediately.

Changed in juju:
status: Triaged → Won't Fix
Revision history for this message
Stuart Bishop (stub) wrote :

I think this is dependent on security bug #892552

Changed in juju-core:
milestone: none → 2.1.0
affects: juju-core → juju
Changed in juju:
milestone: 2.1.0 → none
milestone: none → 2.1.0
Revision history for this message
Menno Finlay-Smits (menno.smits) wrote :

This was fixed a long time ago as part of bug 892552. Bug 1579593 still remains.

Changed in juju:
milestone: 2.1.0 → none
status: Triaged → Fix Released
Tom Haddon (mthaddon)
Changed in juju-deployer:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers