[SRU] eic_harvest_hostkeys fails in local zones

[Robert C Jennings]

[Impact]

* ec2-instance-connect breaks during host key harvesting for instances launched in local zones [1] making the system boot to degraded mode only.

[Test Plan]

* Start a system with the the fixed ec2-instance-connect package in a local zone [1] or break the the /usr/share/ec2-instance-connect/eic_harvest_hostkeys script to exit with failure.

[Where problems could occur]

* The fix is ignoring the eic_harvest_hostkeys script's exit code which may hide actual problems in the script or in the infrastructure preventing connecting to the instance using Instance Connect. This is a decision by upstream. There are no other expected issues.

[Original Bug Text]

ec2-instance-connect breaks during host key harvesting for instances launched in local zones[1]. Here are is the relevant debug data:

$ systemctl is-system-running
degraded

$ systemctl list-units --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
● ec2-instance-connect.service loaded failed failed EC2 Instance Connect Host Key Harvesting

$ journalctl --unit ec2-instance-connect
-- Logs begin at Wed 2021-02-10 22:47:47 UTC, end at Wed 2021-02-10 22:55:46 UTC. --
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Starting EC2 Instance Connect Host Key Harvesting...
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Main process exited, code=exited, status=255/EXCEPTION
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: ec2-instance-connect.service: Failed with result 'exit-code'.
Feb 10 22:48:16 ip-172-31-51-82 systemd[1]: Failed to start EC2 Instance Connect Host Key Harvesting.

$ dpkg-query -l ec2-instance-connect
ii ec2-instance-connect 1.1.13-0ubuntu1 all Configures ssh daemon to accept EC2 Instance Connect ssh keys

$ lsb_release -c
Codename: hirsute

$ cat /etc/cloud/build.info
build_name: server
serial: 20210208

$ ec2metadata --availability-zone --ami-id
us-west-2-lax-1a
ami-098f71a7a25a0f1f2

$ bash -x /usr/share/ec2-instance-connect/eic_harvest_hostkeys
...
++ /usr/bin/curl -s -f -m 1 -H 'X-aws-ec2-metadata-token: AQAEAEvStI0Ugwz1C3GQh7oubFTah7bXQllCmFU6BtMI6b6l5zMkVQ==' http://169.254.169.254/latest/meta-data/placement/availability-zone/
+ zone=us-west-2-lax-1a
+ zone_exit=0
+ '[' 0 -ne 0 ']'
+ /bin/echo us-west-2-lax-1a
+ /bin/grep -Eq '^([a-z]+-){2,3}[0-9][a-z]$'
+ /usr/bin/head -n 1
+ exit 255

[1] https://aws.amazon.com/about-aws/global-infrastructure/localzones/

[Robert C Jennings (rcj)]

This is a problem with "# Validate the zone" logic @ https://git.launchpad.net/ubuntu/+source/ec2-instance-connect/tree/src/bin/eic_harvest_hostkeys?h=applied/ubuntu/hirsute#n92

[Robert C Jennings (rcj)]

And I haven't tested, but this would also fail in Wavelength zones with names in the form `us-east-1-wl1-bos-wlz-1`. Even if instance connect isn't supported in wavelength zones I rather not have the service fail and mark the system status as degraded.

[Robert C Jennings (rcj)]

Wavelength zones are described @ https://aws.amazon.com/wavelength/features/#Locations

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-instance-connect - 1.1.14-0ubuntu1

---------------
ec2-instance-connect (1.1.14-0ubuntu1) hirsute; urgency=medium

  * Bugfix only upload
    Also contains test improvements which don't affect the package in Ubuntu.
  * New upstream version 1.1.13 (LP: #1915345):
    - System startup now ignores failures on host key harvesting.

 -- Balint Reczey <email address hidden> Mon, 29 Mar 2021 20:48:45 +0200

[Brian Murray (brian-murray)]

Hello Robert, or anyone else affected,

Accepted ec2-instance-connect into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-instance-connect/1.1.12+dfsg1-0ubuntu3.20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Robert, or anyone else affected,

Accepted ec2-instance-connect into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-instance-connect/1.1.12+dfsg1-0ubuntu3.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Robert, or anyone else affected,

Accepted ec2-instance-connect into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-instance-connect/1.1.12+dfsg1-0ubuntu3~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robie Basak (racb)]

Hello Robert, or anyone else affected,

Accepted ec2-instance-connect into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ec2-instance-connect/1.1.12+dfsg1-0ubuntu3~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Robert C Jennings (rcj)]

I have tested in AWS local zone us-west-2-lax-1a

xenial:
 image: ami-008b09448b998a562
 build serial: 20201014
 ec2-instance-connect 1.1.12+dfsg1-0ubuntu3~16.04.2

bionic:
 image: ami-02701bcdc5509e57b
 build serial: 20210224
 ec2-instance-connect 1.1.12+dfsg1-0ubuntu3~18.04.2

focal:
 image: ami-0ca5c3bd5a268e7db
 build serial: 20210223
 ec2-instance-connect 1.1.12+dfsg1-0ubuntu3.20.04.1

groovy:
 image: ami-0c1204e0c5e73ef4c
 build serial: 20210325
 ec2-instance-connect 1.1.12+dfsg1-0ubuntu3.20.10.1

Are you sure this is right? Yes, the systemd unit no longer fails because the patch ignored the script failure, but does it *work*? The package update has no changes to /usr/share/ec2-instance-connect/eic_harvest_hostkeys to match the string format for a local zone still. So while the feature is available in local and wavelength zones the package in -proposed fails to address the underlying failure and so ec2-instance-connect still broken in those zones (just silently now).

Again, here is the failure:

$ sudo sh -x /usr/share/ec2-instance-connect/eic_harvest_hostkeys 2>&1 | tail -9
+ /usr/bin/curl -s -f -m 1 -H X-aws-ec2-metadata-token: AQAEAF6AxckVUQFPqe3ivPjLa0b7dlvf4To2TaAReHD-lMpqgvuXBQ== http://169.254.169.254/latest/meta-data/placement/availability-zone/
+ zone=us-west-2-lax-1b
+ zone_exit=0
+ [ 0 -ne 0 ]
+ /bin/echo us-west-2-lax-1b
+ /usr/bin/head -n 1
+ /bin/grep -Eq ^([a-z]+-){2,3}[0-9][a-z]$
+ exit 255
+ rm -rf /dev/shm/eic-hostkey-WZBt1Vck

Please look at the grep on line 101 of the script:
 # Validate the zone
 /bin/echo "${zone}" | /usr/bin/head -n 1 | /bin/grep -Eq "^([a-z]+-){2,3}[0-9][a-z]$" || exit 255

The script needs to handle matches to the existing regex, but also local zones like 'us-west-2-lax-1b' and wavelength zones like 'us-west-2-wl1-den-wlz-1'

[Robert C Jennings (rcj)]

I've left a comment in the upstream bug https://github.com/aws/aws-ec2-instance-connect-config/issues/28#issuecomment-816650597 regarding my SRU test feedback.

[Balint Reczey (rbalint)]

@rcj Thank you for testing the package. I realized that the "fix" is just ignoring the error and noted this regression in [Where problems could occur] section. I, possibly wrongly, assumed that this is upstream's intention and thank you for asking for clarification upstream.

For now I don't change the uploaded packages, but wait for upstream's feedback if this is the solution they really want.

[Balint Reczey (rbalint)]

AWS did not respond on GitHub nor on internal channels to clarify if they are happy with their current fix. I agree with @rcj's concern that usability-wise the fix could be improved, but this improvement can take place later in a separate SRU. I'm marking the bug as verification-done because the fix does what it is expected to do and we received no further information about it.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-instance-connect - 1.1.12+dfsg1-0ubuntu3.20.10.1

---------------
ec2-instance-connect (1.1.12+dfsg1-0ubuntu3.20.10.1) groovy; urgency=medium

  * System startup now ignores failures on host key harvesting (LP: #1915345)

 -- Balint Reczey <email address hidden> Wed, 31 Mar 2021 21:04:31 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for ec2-instance-connect has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-instance-connect - 1.1.12+dfsg1-0ubuntu3.20.04.1

---------------
ec2-instance-connect (1.1.12+dfsg1-0ubuntu3.20.04.1) focal; urgency=medium

  * System startup now ignores failures on host key harvesting (LP: #1915345)

 -- Balint Reczey <email address hidden> Wed, 31 Mar 2021 21:04:31 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-instance-connect - 1.1.12+dfsg1-0ubuntu3~18.04.2

---------------
ec2-instance-connect (1.1.12+dfsg1-0ubuntu3~18.04.2) bionic; urgency=medium

  * System startup now ignores failures on host key harvesting (LP: #1915345)

 -- Balint Reczey <email address hidden> Wed, 31 Mar 2021 21:04:31 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package ec2-instance-connect - 1.1.12+dfsg1-0ubuntu3~16.04.2

---------------
ec2-instance-connect (1.1.12+dfsg1-0ubuntu3~16.04.2) xenial; urgency=medium

  * System startup now ignores failures on host key harvesting (LP: #1915345)

 -- Balint Reczey <email address hidden> Wed, 31 Mar 2021 21:04:31 +0200