Unable to SSH after update and image

[Andy Mahoney]

When creating an image with an OS update (using packer or otherwise) from an Ubuntu image that has google-guest-agent 20210629.00-0ubuntu1~20.04.0 available in the updates repository, the agent does not start on the first boot of the image and therefore it can become impossible to connect via SSH without a reboot.

[Andy Mahoney (ajmahoney)]

Steps to reproduce:
This has been tested on ubuntu-2004-focal-v20210927 and ubuntu-minimal-1804-bionic-v20210915.

Ensure new-user1 and new-user2 do not have ssh keys in the project metadata.

1, Create a new instance and apply OS updates:
gcloud compute instances create ssh-test-image --machine-type=f1-micro --image=ubuntu-2004-focal-v20210927 --image-project=ubuntu-os-cloud --metadata=startup-script=\#\!/bin/bash$'\n'apt-get\ update$'\n'apt-get\ -y\ dist-upgrade --zone=europe-west4-a

gcloud compute instances tail-serial-port-output ssh-test-image --zone=europe-west4-a

Wait for finish e.g. 'Startup finished in X.XXs (kernel) + Xmin XX.XXXs (userspace) = Xmin XX.XXXs.' then press Ctrl+C

2, Stop instance and create an image:
gcloud compute instances stop ssh-test-image --zone=europe-west4-a

gcloud compute images create ssh-test-image --source-disk=ssh-test-image --source-disk-zone=europe-west4-a

3, Create a new instance from the image:
gcloud compute instances create ssh-test-instance --machine-type=f1-micro --image=ssh-test-image --zone=europe-west4-a

gcloud compute instances tail-serial-port-output ssh-test-instance --zone=europe-west4-a

# Wait for finish e.g. 'Startup finished in X.XXs (kernel) + XX.XXXs (userspace) = XX.XXXs.' then press Ctrl+C

4, Attempt to connect with a new user:
gcloud compute ssh new-user1@ssh-test-instance --zone=europe-west4-a
Updating project ssh metadata...⠹Updated [https://www.googleapis.com/compute/v1/projects/kaboodle-labs].
Updating project ssh metadata...done.
Waiting for SSH key to propagate.
Warning: Permanently added 'compute.9106208184098581676' (ECDSA) to the list of known hosts.
new-user1@34.141.231.113: Permission denied (publickey).
new-user1@34.141.231.113: Permission denied (publickey).
new-user1@34.141.231.113: Permission denied (publickey).
new-user1@34.141.231.113: Permission denied (publickey).
new-user1@34.141.231.113: Permission denied (publickey).
new-user1@34.141.231.113: Permission denied (publickey).
new-user1@34.141.231.113: Permission denied (publickey).
new-user1@34.141.231.113: Permission denied (publickey).
new-user1@34.141.231.113: Permission denied (publickey).
new-user1@34.141.231.113: Permission denied (publickey).
new-user1@34.141.231.113: Permission denied (publickey).
ERROR: (gcloud.compute.ssh) Could not SSH into the instance. It is possible that your SSH key has not propagated to the instance yet. Try running this command again. If you still cannot connect, verify that the firewall and instance are set to accept ssh traffic.

5, Restart the instance and attempt to connect again
gcloud compute instances stop ssh-test-instance --zone=europe-west4-a

gcloud compute instances start ssh-test-instance --zone=europe-west4-a

gcloud compute ssh new-user1@ssh-test-instance --zone=europe-west4-a
Welcome to Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-1020-gcp x86_64)

 * Documentation: https://help.ubuntu....