Key trust verification fails on Ubuntu 22.04

[Utkarsh Gupta]

[Impact]
========

Can't get EC2 instance connect to work on Ubuntu 22.04 due to what seems to be an issue with trust chain verification. This is due to a change in OpenSSL 3.0.2.

[Test Plan]
===========

To reproduce:

- Launch an EC2 instance with the current Ubuntu 22.04 AMI (e.g. ami-0aeb7c931a5a61206 in us-east-2).
- Try to connect to it via mssh ubuntu@<instance-id>.
- Observe that the command fails with "Permission denied (publickey)."

When using the --debug flag with mssh, I see that the public key is pushed successfully, but the remote rejects the connection:

```
...
2022-05-06 09:10:58,549 - EC2InstanceConnect - DEBUG - Successfully got instance information from EC2 API for <instance-id>
...
2022-05-06 09:10:59,189 - EC2InstanceConnect - DEBUG - Successfully pushed the public key to <instance-id>
2022-05-06 09:10:59,190 - EC2InstanceConnect - DEBUG - Generated command: ssh -o "IdentitiesOnly=yes" -i /var/folders/30/xdglsm2j3tz1rn1n7yygtm7c0000gn/T/tmp33a253uf ubuntu@<ip>
ubuntu@<ip>: Permission denied (publickey).
2022-05-06 09:10:59,612 - EC2InstanceConnect - DEBUG - Deleting the private key file: /var/folders/30/xdglsm2j3tz1rn1n7yygtm7c0000gn/T/tmp33a253uf
```

On the instance side, the following error is logged:

```
AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:wiFxouWj6qQ0aUZ0CAcftWZqNEf3qj2LLicCfGFcQJY failed, status 2
```

[Where Problems Could Occur]
============================

The package is broken for 22.04 so not a lot of things can go wrong there. However, if the user has done some manual workarounds, it could break that. But chances are less, IMO. \o/

[Upstream Bug and Fix]
======================

https://github.com/aws/aws-ec2-instance-connect-config/issues/38
https://github.com/aws/aws-ec2-instance-connect-config/pull/39

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-instance-connect - 1.1.14-0ubuntu2

---------------
ec2-instance-connect (1.1.14-0ubuntu2) kinetic; urgency=medium

  * Add patch to fix parse authorized keys script
    to work with OpenSSL 3.0.2. (LP: #1973114)

 -- Utkarsh Gupta <email address hidden> Thu, 12 May 2022 10:58:05 +0530

[Chris Newcomer (cnewcomer)]

Validation of package in Kinetic. The validation was run on Jammy with the Kinetic package installed:

I have confirmed the bug on a 22.04 image running on AWS:
Jammy stock ec2-instance-connect ssh -vvv output:
https://pastebin.ubuntu.com/p/jDCnKrGRFM/plain/

When upgrading the package on the same instance with the one from Kinetic (with backported fix), it works:
Jammy upgraded (kinetic) ec2-instance-connect ssh -vvv output:
https://pastebin.ubuntu.com/p/bkvGwzZDYB/plain/

[Fabio Augusto Miranda Martins (fabio.martins)]

I also tested with mssh to make it works well:

When running ec2-instance-connect 1.1.14-0ubuntu1 (from Jammy) and trying to connect with mssh:

fabio@fabio-canonical:~/.aws$ mssh ubuntu@i-<ami_id>
ubuntu@<ip>: Permission denied (publickey).

Due to the bug, we can see the key being pushed, but then the client denying the connection:

https://pastebin.ubuntu.com/p/7StqsfQdJp/

Back in the instance, I've changed the sources.list to point to kinetic repos and upgraded ec2-instance-connect to kinetic's version:

ubuntu@ip-10-0-1-226:~$ sudo apt-cache policy ec2-instance-connect
ec2-instance-connect:
  Installed: 1.1.14-0ubuntu2
  Candidate: 1.1.14-0ubuntu2
  Version table:
 *** 1.1.14-0ubuntu2 500
        500 http://us-east-1.ec2.archive.ubuntu.com/ubuntu kinetic/main amd64 Packages
        100 /var/lib/dpkg/status

And now mssh works fine:

fabio@fabio-canonical:~/.aws$ mssh ubuntu@i-<ami_id>
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-1004-aws x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

  System information as of Fri May 13 14:16:20 UTC 2022

  System load: 0.2421875 Processes: 128
  Usage of /: 21.1% of 7.58GB Users logged in: 1
  Memory usage: 1% IPv4 address for eth0: 10.0.1.226
  Swap usage: 0%

74 updates can be applied immediately.
To see these additional updates run: apt list --upgradable

Last login: Fri May 13 13:47:05 2022 from <ip>
ubuntu@ip-10-0-1-226:~$

[Brian Murray (brian-murray)]

Hello Utkarsh, or anyone else affected,

Accepted ec2-instance-connect into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-instance-connect/1.1.14-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Fabio Augusto Miranda Martins (fabio.martins)]

Hi Brian,

Thanks for that. I've tested and validated that ec2-instance-connect from jammy-proposed fixes the issue. Here are the evidences:

https://pastebin.ubuntu.com/p/dPD6vyS6g4/

Cheers,
Fabio Martins

[Andrew Cloke (andrew-cloke)]

Thanks Fabio. Adjusted tags accordingly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-instance-connect - 1.1.14-0ubuntu1.1

---------------
ec2-instance-connect (1.1.14-0ubuntu1.1) jammy; urgency=medium

  * Add patch to fix parse authorized keys script
    to work with OpenSSL 3.0.2. (LP: #1973114)

 -- Utkarsh Gupta <email address hidden> Thu, 12 May 2022 10:59:05 +0530

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for ec2-instance-connect has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.