ssh daemon hangs after publickey packet sent

[Scott Moser]

A launched ec2 instance in ap-southeast-1 region is unreachable via ssh.
$ ssh -vvv ec2-175-41-171-225.ap-southeast-1.compute.amazonaws.com
shows progress up to :

debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: smoser@brickies
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply

Then nothing for minutes before session is killed (manually).

In a 'good' connection, the following would be next:
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey

I'll attach full 'ssh -vvv' logs good and bad connection.

ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: openssh-server 1:5.5p1-4ubuntu3
ProcVersionSignature: User Name 2.6.35-14.19-virtual 2.6.35
Uname: Linux 2.6.35-14-virtual x86_64
Architecture: amd64
Date: Tue Aug 3 14:45:25 2010
Ec2AMI: ami-9fc4bbcd
Ec2AMIManifest: ubuntu-images-testing-ap-southeast-1/ubuntu-maverick-daily-amd64-server-20100803.1.manifest.xml
Ec2AvailabilityZone: ap-southeast-1a
Ec2InstanceType: m1.large
Ec2Kernel: aki-11d5aa43
Ec2Ramdisk: unavailable
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: openssh

[Scott Moser (smoser)]

Other information:
ssh-keyscan did show results:

# ec2-175-41-171-225.ap-southeast-1.compute.amazonaws.com SSH-2.0-OpenSSH_5.5p1 Debian-4ubuntu3
ec2-175-41-171-225.ap-southeast-1.compute.amazonaws.com ssh-rsa AAAAB3NzaC1y...RzIcF9UuISZaYJf

A reboot-instances seems to have no affect.

What is strange here is that there was a key in ssh-keyscan, yet the console shows no key-gen output. This is likely related to the fact that the instance would not reboot.

I suspect kernel problem.

[Scott Moser (smoser)]

The reboot-instances did eventually occur (it did not have immediate effect), and after, it came up fine.
I'm attaching the post-reboot log here.

It contains messages like:
[ 240.170158] INFO: task sshd:668 blocked for more than 120 seconds.^M
[ 240.170175] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.^M
[ 360.170086] INFO: task sshd:668 blocked for more than 120 seconds.^M
[ 360.170102] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.^M

[Scott Moser (smoser)]

I've seen this again now, on:
eu-west-1 ami-53dbf127 canonical ubuntu-maverick-daily-amd64-server-20100803.1

[Scott Moser (smoser)]

Marking this wishlist, it appears very uncommon and i really think its either ssh client or server that has the issue.

[Ben Howard (darkmuggle-deactivatedaccount)]

Scott -- are the instances that exhibit this bug EBS-backed or S3-backed?

[Scott Moser (smoser)]

Well, the two ami ids listed above (in the bug opening and then comment 7) were both instance-store.

[Scott Moser (smoser)]

marking invalid on cloud-init, definitely the bug was not/is-not here.

[Ben Howard (darkmuggle-deactivatedaccount)]

Closing this out as crufty. If this is seen again, please reopen.

[Graham Leggett (minfrin-y)]

I am seeing this bug in Ubuntu v14.04. No obvious cause. When it's happened we've physically replaced the instances, as there is no console access at AWS.

[Atti (tennov)]

i see the same in Ubuntu 18.04.
with VPN ssh hangs on message

"debug2: we sent a publickey packet, wait for reply"

without VPN terminal itself hangs on message

"debug1: pledge: network"