jammy daily won't accept EC2 publickey (suspect cloud-init failure)

[Liz Fong-Jones]

amazon-ebs: output will be in this color.

==> amazon-ebs: Prevalidating any provided VPC information
==> amazon-ebs: Prevalidating AMI Name: packer-honeycomb-arm-1649438444
    amazon-ebs: Found Image ID: ami-03c980393156217bb
==> amazon-ebs: Creating temporary keypair: packer_62506eec-8c01-8c21-b199-32a96021bd7a
==> amazon-ebs: Creating temporary security group for this instance: packer_62506eee-c3da-1eb2-6e81-baa0c5ccfdfc
==> amazon-ebs: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebs: Launching a source AWS instance...
==> amazon-ebs: Adding tags to source instance
    amazon-ebs: Adding tag: "Name": "Packer Builder"
    amazon-ebs: Instance ID: i-01158f222c8664f0c
==> amazon-ebs: Waiting for instance (i-01158f222c8664f0c) to become ready...
==> amazon-ebs: Using SSH communicator to connect: 54.173.181.221
==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Error waiting for SSH: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: Cleaning up any extra volumes...
==> amazon-ebs: No volumes to clean up, skipping
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored after 2 minutes 42 seconds: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

==> Wait completed after 2 minutes 42 seconds

==> Some builds didn't complete successfully and had errors:
--> amazon-ebs: Packer experienced an authentication error when trying to connect via SSH. This can happen if your username/password are wrong. You may want to double-check your credentials as part of your debugging process. original error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

==> Builds finished but no artifacts were created.

[James Falcon (falcojr)]

Have you tried this on non-jammy instance, and if so, does it work there? Do you have a packer and/or cloud-init script that you used, or a stripped-down version that could be used to reproduce the issue?

[Liz Fong-Jones (lizthegrey)]

Yes, it works fine with a bionic or focal base image.

Yes, I can send along the packer script but it's really really basic, just specifies an ARM instance type and to use the ssh communicator (and then will attempt to install cinc/chef and run the cookbooks, but it's failing before it's able to get there).

[Liz Fong-Jones (lizthegrey)]

Focal install log:

==> amazon-ebs: Waiting for instance (i-0edef3c6221f38b5b) to become ready...
==> amazon-ebs: Using SSH communicator to connect: 3.88.16.124
==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Connected to SSH!
==> amazon-ebs: Provisioning with chef-solo
    amazon-ebs: Installing Chef...
[...]

[Liz Fong-Jones (lizthegrey)]

bisecting the problem: impish ami-0a4b1fdc2301bf1f1 does not have this problem either. so it's something that broke between impish and jammy.

[Liz Fong-Jones (lizthegrey)]

var-file:
{
  "ami_name": "honeycomb-arm",
  "instance_type": "c6g.large",
  "source_ami": "ami-03c980393156217bb"
}

packer.json:
{
  "builders": [
    {
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "{{user `region`}}",
      "source_ami": "{{user `source_ami`}}",
      "instance_type": "{{user `instance_type`}}",
      "ssh_username": "ubuntu",
      "ena_support": true,
      "sriov_support": true,
      "launch_block_device_mappings": [
        {
          "device_name": "/dev/sda1",
          "volume_type": "gp2",
          "volume_size": 30,
          "delete_on_termination": true
        }
      ],
      "ami_name": "packer-{{user `ami_name`}}-{{timestamp}}",
      "ami_description": "ubuntu 22.04 on {{user `instance_type`}} based on {{user `source_ami`}}, Jammy LTS updated on {{isotime \"2006-01-02\"}}",
      "tags": {
        "AMI": "{{user `ami_name`}}",
        "SHA": "{{user `ami_sha`}}"
      }
    }
  ],
  "provisioners": [
    {
      "type": "chef-solo",
      "install_command": "wget http://downloads.cinc.sh/files/stable/cinc/17.10.0/ubuntu/20.04/cinc_17.10.0-1_arm64.deb && sudo dpkg -i cinc_17.10.0-1_arm64.deb",
      "cookbook_paths": [
        "../chef/cookbooks",
        "../chef/vendored_cookbooks"
      ],
      "data_bags_path": "../chef/data_bags",
      "run_list": [
        "hound_base::packer"
      ]
    }
  ],
  "variables": {
    "ami_name": "",
    "ami_sha": "",
    "instance_type": "",
    "source_ami": "",
    "region": ""
  }
}

[James Falcon (falcojr)]

The AMI referenced there doesn't seem to be valid, and I can't otherwise reproduce the behavior. There's no difference in SSH handling that I can see between Impish and Jammy, so I'm not sure what else can be done here.

[Liz Fong-Jones (lizthegrey)]

ami-03c980393156217bb is listed on https://cloud-images.ubuntu.com/locator/daily/ as the image for com.ubuntu.cloud.daily:server:22.04 20220407 arm64 in us-east-1.

[Liz Fong-Jones (lizthegrey)]

I've checked all of the following jammy daily images, and all of them fail to correctly install the EC2 metadata supplied SSH key such that it is possible to ssh ubuntu@host after startup.

Amazon AWS us-east-1 jammy com.ubuntu.cloud.daily:server:22.04 arm64 hvm-ssd 20220323 ami-0320e0e27a2e26467
Amazon AWS us-east-1 jammy com.ubuntu.cloud.daily:server:22.04 arm64 hvm-ssd 20220405 ami-028c98d9274336455
Amazon AWS us-east-1 jammy com.ubuntu.cloud.daily:server:22.04 arm64 hvm-ssd 20220406 ami-0cb3473d91f8f239d
Amazon AWS us-east-1 jammy com.ubuntu.cloud.daily:server:22.04 arm64 hvm-ssd 20220407 ami-03c980393156217bb

Both of these do install the SSH key for ubuntu@ from AWS metadata fine:
Amazon AWS us-east-1 impish 21.10 arm64 hvm-ssd 20220309 ami-0a4b1fdc2301bf1f1
Amazon AWS us-east-1 impish com.ubuntu.cloud.daily:server:21.10 arm64 hvm-ssd 20220319 ami-09ce10196abcc5dc4

[Liz Fong-Jones (lizthegrey)]

https://github.com/hashicorp/packer/issues/11656#issuecomment-1076175764

found my problem. packer generates temporary ssh-rsa keys which are disallowed by the 22.04 default SSH key policy.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3967