Getting locked out of long running GCE instances using juju ssh & juju scp
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Fix Released
|
Critical
|
Witold Krecicki | ||
cloud-images |
Invalid
|
Undecided
|
Unassigned |
Bug Description
Encountered a problem with connecting to long running GCE instance and only using juju scp and juju ssh to connect to this VM. I believe juju is doing something irregular that is getting my IP address blocked from connecting to the VM.
I was actively using a juju ssh session to jenkins/1 in one terminal, and trying to scp a file up to the jenkins/1 in another terminal. I was able to connect earlier and transfer a file down (using juju scp). When trying to upload the file I got disconnected and it appears my IP address is blocked. Here is the actual commands I was running and what happened in the terminal:
$ juju scp juju_gce.tar.gz jenkins/
ERROR exit status 1 (Timeout, server 104.197.80.216 not responding.
lost connection)
$ juju scp juju_gce.tar.gz jenkins/1:
ERROR cannot connect to any address: [104.197.80.216:22 10.240.0.3:22 172.17.0.1:22]
$ juju ssh jenkins/1
ERROR cannot connect to any address: [104.197.80.216:22 10.240.0.3:22 172.17.0.1:22]
$ ping 104.197.80.216
PING 104.197.80.216 (104.197.80.216) 56(84) bytes of data.
^C
--- 104.197.80.216 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1031ms
The terminal using juju ssh timed out at the same time I got this error:
Timeout, server 104.197.80.216 not responding.
- - - - -
After these error messages I am unable to juju scp or juju ssh to this server and I can not connect with ping or regular ssh.
I was blocked out at my home address, so I traveled to a coffee shop where was able to connect for a short time before getting blocked again. It appears to me that Juju is doing something irregular and getting blocked on GCE instances.
Please help with this problem.
description: | updated |
Changed in juju: | |
status: | Triaged → Incomplete |
Changed in juju: | |
importance: | High → Critical |
status: | Incomplete → Triaged |
assignee: | nobody → John A Meinel (jameinel) |
milestone: | none → 2.2-beta3 |
milestone: | 2.2-beta3 → 2.2-rc1 |
Changed in juju: | |
assignee: | John A Meinel (jameinel) → Witold Krecicki (wpk) |
Changed in juju: | |
milestone: | 2.2-beta4 → 2.2-rc1 |
Changed in juju: | |
status: | In Progress → Fix Committed |
Changed in juju: | |
status: | Fix Committed → Fix Released |
Changed in cloud-images: | |
status: | New → Invalid |
I am able to access the GCE VM using the Google console. I see my IP addresses in the iptable --lis DROP rules! The coffee shop ip address is 96.33.226.42.
Chain sshguard (1 references) planet- telecom. eu anywhere 208-112. myvps.jp anywhere 45.dhcp. roch.mn. charter. com anywhere 86.109. triolan. net anywhere gocheepmobile. com anywhere 219-77. rev.poneyteleco m.eu anywhere 188.dhcp. roch.mn. charter. com anywhere
target prot opt source destination
DROP all -- smtp1.orderz.com anywhere
DROP all -- hostby.
DROP all -- 122.194.229.9 anywhere
DROP all -- 112.85.42.110 anywhere
DROP all -- v157-7-
DROP all -- 153.99.182.8 anywhere
DROP all -- 112.85.42.28 anywhere
DROP all -- 112.85.42.22 anywhere
DROP all -- 193.201.224.237 anywhere
DROP all -- 96-42-224-
DROP all -- 123.183.209.136 anywhere
DROP all -- 106.226.
DROP all -- 146.228.112.199 anywhere
DROP all -- 114.119.7.53 anywhere
DROP all -- fs.sip.
DROP all -- 120.27.133.147 anywhere
DROP all -- 123.183.209.135 anywhere
DROP all -- 163-172-
DROP all -- 96-42-209-
DROP all -- 117.54.13.180 anywhere
DROP all -- 91.224.160.131 anywhere
DROP all -- 46.148.18.163 anywhere
It looks like Juju scp/ssh is doing something that sets off a sshguard rule on GCE VMs. Here is the output from the systemctl status sshguard
Mar 02 15:55:35 juju-25c78b-1 sshguard[1710]: Blocking 96.33.226.42:4 for >630secs: 40 danger in 4 attacks over 907 seconds (all: 40d in 1 abuses over 907s).
The only method of connection was juju ssh/scp to connect to this system and now my IP address is blocked!