cloud-init cloud-config for ssh broken in jammy

[Thomas Bechtold]

[impact]
cloud-config overrides via cloud-init for ssh do no longer work. Eg.

#cloud-config
ssh_pwauth: True

doesn't enable password authentication anymore.

The reason is that https://git.launchpad.net/livecd-rootfs/commit/live-build/ubuntu-cpc/hooks.d/chroot/052-ssh_authentication.chroot?h=ubuntu/jammy&id=3b2eeb017153cbbfc8935f9447e00d196e34d983 is a wrong backport using the wrong filename /etc/ssh/sshd_config.d/10-cloudimg-settings.conf . it needs to be /etc/ssh/sshd_config.d/60-cloudimg-settings.conf . See https://git.launchpad.net/livecd-rootfs/tree/live-build/ubuntu-cpc/hooks.d/chroot/052-ssh_authentication.chroot?h=ubuntu/master which is the correct change in the ubuntu/master branch.

This bug needs only fixing in Jammy and Focal because the fix is correct in > Jammy. Jammy has this bug already in the -updates pocket but Focal only has the wrong fix in its git branch.

[test plan]
* build an image with the proposed fix
* use a cloud-init cloud-config data to enable ssh password auth (which is disabled by default)
* check after image boot with "sshd -T" if password authentication is enabled
* make sure that the file written by cloud-init in /etc/ssh/sshd_config.d/ has a lower number than the file from the image (cloudimg-settings.conf)

[ Where problems could occur ]
The change does rename a single file to have correct ordering between the prebuilt cloud images and cloud-init. There are edge-cases where this could create problems but those are acceptable:

- rebuilt images based on the Ubuntu image which try to delete 10-cloudimg-settings.conf would no longer work

But in general this is a bug fix for the changes introduced in LP:#1968873

[Steve Langasek (vorlon)]

An upload of livecd-rootfs to jammy-proposed has been rejected from the upload queue for the following reason: "needs to be accompanied in same SRU cycle with fix for LP: #2049723".

[Andreas Hasenack (ahasenack)]

I see a livecd-rootfs upload to jammy-unapproved, so adding a jammy task. But I also see an MP against focal linked to this bug, do we also need this fixed in focal?

[Andreas Hasenack (ahasenack)]

> Jammy has this bug already in the -updates pocket but Focal only has the wrong fix in its git branch.

Ah, so focal was never released. Ok.

[Steve Langasek (vorlon)]

Hello Thomas, or anyone else affected,

Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.765.38 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Thomas Bechtold (toabctl)]

Verified that it works via:

- test image build for EC2
- launching a instance with user-data to set password authentication: "aws ec2 run-instances --image-id ami-01046735e1a43b710 --instance-type m6a.large --key-name toabctl --user-data file:///home/tom/tmp/cloud-config --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=toabctl-jammy-sru-validation}]'"
- login with password: ssh -o PubkeyAuthentication=no ubuntu@3.66.190.126

Ordering looks good now:

# ls -al /etc/ssh/sshd_config.d/
total 16
drwxr-xr-x 2 root root 4096 Feb 1 14:23 .
drwxr-xr-x 4 root root 4096 Feb 1 14:23 ..
-rw------- 1 root root 27 Feb 1 14:23 50-cloud-init.conf
-rw-r--r-- 1 root root 26 Feb 1 13:44 60-cloudimg-settings.conf

# sshd -T|grep passwordauth
passwordauthentication yes

[Timo Aaltonen (tjaalton)]

Hello Thomas, or anyone else affected,

Accepted livecd-rootfs into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.664.52 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.765.38

---------------
livecd-rootfs (2.765.38) jammy; urgency=medium

  * Add a largemem subarch for ubuntu-server that ships a 64k kernel variant
    by default (LP: #2050209)

livecd-rootfs (2.765.37) jammy; urgency=medium

  * fix: Fix for calling unminimize if lxd-installer package
    not installed. (LP: #2049723)

livecd-rootfs (2.765.36) jammy; urgency=medium

  * Use correct /etc/ssh/sshd_config.d/ filename so cloud-init overrides
    via cloud-config works. (LP: #2049860)

 -- Łukasz 'sil2100' Zemczak <email address hidden> Thu, 25 Jan 2024 11:46:49 +0100

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Thomas Bechtold (toabctl)]

Verified that it works via:

- test image build for EC2 for Focal
- launching a instance with user-data to set password authentication: "aws ec2 run-instances --image-id ami-00ba68cfdb7ccee60 --instance-type m6a.large --key-name toabctl --user-data file:///home/tom/tmp/cloud-config --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=toabctl-focal-sru-validation}]'"
- login with password: ssh -o PubkeyAuthentication=no ubuntu@3.68.190.249

Ordering looks good now:

# ls -al /etc/ssh/sshd_config.d/
total 16
drwxr-xr-x 2 root root 4096 Feb 8 11:19 .
drwxr-xr-x 4 root root 4096 Feb 8 11:19 ..
-rw------- 1 root root 27 Feb 8 11:19 50-cloud-init.conf
-rw-r--r-- 1 root root 26 Feb 8 10:47 60-cloudimg-settings.conf

# sshd -T|grep passwordauth
passwordauthentication yes

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.664.52

---------------
livecd-rootfs (2.664.52) focal; urgency=medium

  * fix: use correct sshd_config.d/ ordering. (LP: #2049860)

livecd-rootfs (2.664.51) focal; urgency=medium

  [ Steve Langasek ]
  * The chroot tmpfs mount should only be /var/lib/apt/lists, not
    /var/lib/apt; the latter breaks changes to /var/lib/apt/extended_states.
    (LP: #2036195).

livecd-rootfs (2.664.50) focal; urgency=medium

  * Do not modify /etc/ssh/sshd_config for ubuntu-cpc
    project builds. (LP: #1968873)

 -- Thomas Bechtold <email address hidden> Mon, 22 Jan 2024 17:08:05 +0530

[DisasteR (benj.saiz)]

Hi,

Looks like this bug is still present in daily Jammy cloud images.
Any idea when this will be available in new daily builds ?

Regards,

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[John Chittum (jchittum)]

@DisasteR -- could you be more specific? Which images are you seeing this in? which cloud, which download from `cloud-images.ubuntu.com`?

[DisasteR (benj.saiz)]

@jchittum --

I use Jammy images from cloud-images.ubuntu.com.
I tested multiple daily images from 2024 like :
20240315.1
20240319

And i cannot connect to the vm by ssh using password with `ssh_pwauth: True`

Cloud init configuration still set `passwordauthentication no` in `/etc/ssh/sshd_config.d/`

I reverted to image release-20231211 based on this issue date and everything is working fine.

I suppose this is the same problem as status is still `Fix Committed` for cloud-images.

[John Chittum (jchittum)]

Tested working on the image from http://cloud-images.ubuntu.com/releases/jammy/release-20240319/

$ ssh -o "UserKnownHostsFile=/dev/null -o CheckHostIP=no StrictHostKeyChecking no" jchittum@0.0.0.0 -p 2222
The authenticity of host '[0.0.0.0]:2222 ([0.0.0.0]:2222)' can't be established.
ED25519 key fingerprint is <REDACTED>
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[0.0.0.0]:2222' (ED25519) to the list of known hosts.
jchittum@0.0.0.0's password:
Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-101-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/pro

  System information as of Mon Apr 1 11:55:56 UTC 2024

$ ls /etc/ssh/sshd_config.d/
50-cloud-init.conf 60-cloudimg-settings.conf

$ sudo cat /etc/ssh/sshd_config.d/50-cloud-init.conf
PasswordAuthentication yes

$ sudo cat /etc/ssh/sshd_config.d/60-cloudimg-settings.conf
PasswordAuthentication no

####

cloud-init and passwords is a bit confusing. here is a working example of a cloud-init:

#cloud-config
ssh_pwauth: true
users:
    - name: jchittum
      groups: [adm, lxd, sudo]
      passwd: <HASHED_PASSWORD>
      sudo: ALL=(ALL) NOPASSWD:ALL
      shell: /bin/bash
      lock_passwd: false
    - name: timmy
      groups: [adm, lxd, sudo, cdrom, dip]
      ssh_import_id: lp:jchittum
      sudo: ALL=(ALL) NOPASSWD:ALL
      passwd: <HASHED_PASSWD>
      shell: /bin/bash
      lock_passwd: false

NOTES:

passwd was set by running : mkpasswd --method=SHA-512 --rounds=500000
lock_passwd: false is _required_ to make this work. otherwise providing a password won't do anything, and you'll never be able to log in.

I'm wondering if it's from a different version of cloud-init instead?

20231211:
cloud-init 23.3.3-0ubuntu0~22.04.1

20240319:
cloud-init 23.4.4-0ubuntu0~22.04.1

i don't see anything related in the changelog[https://github.com/canonical/cloud-init/blob/main/ChangeLog]

next steps: try a coud-init config like i have above. If it fails, please provide your cloud-init and outputs (especially helpful with some cloud-init logs).

[DisasteR (benj.saiz)]

@jchittum

I setup user with the same parameters and i also generate passwords with mkpasswd --method=SHA-512 but i cannot ssh using password on the listed images.

It work as expected with the old one release-20231211.

I looked the log but didn't find any error.
And password is working locally.

I will made some more test.

[DisasteR (benj.saiz)]

Hi @jchittum

I made some test today using the daily build from 20240403 and same problem.

cloud init version is 23.4.4-0ubuntu0~22.04.1

ssh_pwauth is present in /var/run/cloud-init/combined-cloud-config.json and value is true.

root@r-bsz-test-001:/# grep ssh_pwauth /var/run/cloud-init/combined-cloud-config.json
 "ssh_pwauth": true,

no 50-cloud-init.conf file is created in /etc/ssh/sshd_config.d/

root@r-bsz-test-001:/# ls /etc/ssh/sshd_config.d/
60-cloudimg-settings.conf
root@r-bsz-test-001:/# grep -H PasswordAuthentication /etc/ssh/sshd_config.d/*
/etc/ssh/sshd_config.d/60-cloudimg-settings.conf:PasswordAuthentication no

No errors in /var/log/cloud-init.log

Every other configuration is working fine.

[John Chittum (jchittum)]

@DisatesR : if you're not seeing a 50-cloud-init.conf file, it indicates to me that cloud-init is failing to parse the cloud_init configuration properly, and thus not adding the required configuration to /etc/ssh/sshd_config.d/

could you provide your entire cloud config? you can also use cloud-init to verify your user-data

https://cloudinit.readthedocs.io/en/latest/howto/debug_user_data.html

[Alberto Contreras (aciba)]

Hi @DisatesR.

In addition to John's comment, collecting and sharing cloud-init's logs[1] (making sure no sensitive data is attached) might help us to triage what's happening.

[1]https://cloudinit.readthedocs.io/en/latest/howto/bugs.html#collect-logs

[DisasteR (benj.saiz)]

I verified it and everything is ok.
Here is the collected logs.

[Alberto Contreras (aciba)]

Hi DisasteR, thanks for attaching the logs.

ssh_pwauth gets handled by the set_passwords config module, which runs as part of the cloud-config stage.

The problem is that cloud-config and cloud-final stages didn't run because cloud-init is instructed to be disabled.

Some stanza in your user-data, vendor-data, instance-data or system-cfg instructs the config module write_files, which runs before cloud-config, to disable cloud-init.

One can see this in the log files and in:

# /run/cloud-init/combined-cloud-config.json
...
 "write_files": [
  {
   "owner": "root:root",
   "path": "/etc/cloud/cloud-init.disabled",
   "permissions": "0644"
  }

One solution, if the disablement is still desired, is to use the defer subkey[1] of write-files to instruct cloud-init to touch the file as part of the cloud-final stage, not disturbing the other stages during the first boot.

[1] https://docs.cloud-init.io/en/latest/reference/modules.html#write-files

[DisasteR (benj.saiz)]

I'll try that.
but that worked with previous cloud-init versions -_- is that a new option ?

[DisasteR (benj.saiz)]

@aciba I can confirm that adding `defer: true` on my write_file resolve the problem.

Can you explain if this is caused by a change in cloud-init or if i was just lucky from the start .. ?

[Alberto Contreras (aciba)]

DisasteR: difficult to say if it was a change from the cloud-images, cloud-init or other package without access to the jammy instance. But in any case, write_files, bootcmd and runcmd are config modules that can break the system in multiple ways as they give total control to the user and they have to be used with care.

I do not think Ubuntu can ensure backwards / forwards compatibility of those cloud-init modules in every case.