ec2-instance-connect fails with cert validation on ubuntu 22.04

[Jay Berkenbilt]

If needed, I can provide more exact steps to reproduce this, but hopefully this will be sufficient. Note that follow identical steps with Ubuntu 20.04 results in a working configuration.

Launch an ec2 instance using the latest version of the Ubuntu AMI as returned by this query:

aws ec2 describe-images --filters Name=architecture,Values=x86_64 Name=virtualization-type,Values=hvm Name=name,Values="ubuntu/images/*22.04-amd64-server-*" Name=block-device-mapping.volume-type,Values=gp2 --owners 099720109477

At this moment, that is ami-09db26f1ef0a9f406 in my region, us-east-1.

Send public key:

aws ec2-instance-connect send-ssh-public-key --availability-zone us-east-1a --instance-id i-abcdexample --instance-os-user ubuntu --ssh-public-key file:///home/user/.ssh/id_rsa.pub

(Note: results are identical with .ssh/id_ed25519.pub)

Attempt ssh ubuntu@ip-addr

On the instance, /var/log/auth.log reports a failure.

May 25 18:57:25 ip-10-98-1-66 sshd[1549]: AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:abcdefgexample failed, status 2

Running the failed command as root on the instance shows:

C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
error 89 at 4 depth lookup: Basic Constraints of CA cert not marked critical
C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
error 92 at 4 depth lookup: CA cert does not include key usage extension
error /dev/shm/eic-7MlPua7W/cert.pem: verification failed

I'm not sure where this certificate comes from, what's enforcing the key usage extension, etc. I haven't investigated further other than to verify that it's the same whether I use my RSA key or my ed25519 key (in fact, either way, my ssh client offers both keys, I see two log messages, and they both fail the same way) and to verify that it does work on Ubuntu 20.04. Also tried: apt update; apt dist-upgrade; reboot to ensure everything is up to date, verifying that ca-certificates is installed.

If I use a keypair, I can log in just fine. To reproduce this for above, I launched the instance with a key pair, then moved .ssh/authorized_keys out of the way to see the failure.

Please let me know if there's any other information I should supply or anything else you would like me to try.

[Jay Berkenbilt (ejb)]

This is no longer reproducible.

[Thomas Bechtold (toabctl)]

closing because of comment#1