Cloud-init does not initialize systemd machine id

[Greg Fefelov]

Systemd uses /etc/machine-id file to uniquely identify the machine and to provide machine ID to D-Bus. This file is generated with `systemd-machine-id-setup` command that should be run on system installation (or at least when a system is provisioned). However, cloud-init does not do this.

As far as I can see, all Wily Werewolf VMs provisioned in EC2 have the same machine-id ff1bed6402c14c57ac03a074cdb7479e.

[Scott Moser (smoser)]

Hi,
This does look like it would be useful.
It seems like the thing I should do is :
  once per instance remove /etc/machine-id and run systemd-machine-id-setup

Ideally i think is that any system where the instance-id is a uuid, we could just populate /etc/machine-id with that instance-id.
What wasnt clear was whether or not this would be sufficient. systemd-machine-id-setup does not indicate the proper way to provide a machine id other than letting it randomly generate one.

[Scott Moser (smoser)]

Is it enough during boot to just:
  rm /etc/machine-id
  echo "<uuid>" > /etc/machine-id
?

Note that systemd-firstboot.service is not currently in ubuntu (xenial)

[Greg Fefelov (gfv)]

Unfortunately, systemd reads /etc/machine-id prior to launching any tasks, so I think this not a cloud-init issue. The proper solution, I guess, would be to delete /etc/machine-id before making an ami out of a running instance. Sorry to bother :(

[Greg Fefelov (gfv)]

(systemd generates a random machine-id if there is no /etc/machine-id whatsoever.)

[Scott Moser (smoser)]

Yeah, likely this happens to early in boot for cloud-init to affect.
I've marked livecd-rootfs as a target, which is (i think) proper for changing the images themselves.

[Scott Moser (smoser)]

Greg,
So digged a bit further, it seems that the ubuntu images do the right thing in that /etc/machine-id is not present in the image itself. I suspect your instances that you noticed this on were snapshotted from a once-booted instance or in some other way booted. If that is the case, it would seem like the best advice would be for you to just remove the file before snapshotting an instance for re-use, so that new instances would get a new id generated.

Ideally in my mind, cloud-init or systemd or somethign would realize that this is a new instance and do the right thing, but that seems non-trivial at the moment. so i'll leave this as a wishlist bug on cloud-init.

Thanks.
Scott

[Oliver Grawert (ogra)]

you might want to consider an initrd script that kicks in after mounting the rootfs but before running run-init to switch to the filesystem.

[Alberto Contreras (aciba)]

I think cloud-init's unique responsibility in respect to this matter is to ensure that an instance has a machine-id.

In systemd systems, this is automatically handled by systemd itself [1]:

> When a machine is booted with systemd(1) the ID of the machine will be established. If systemd.machine_id= or --machine-id= options (see first section) are specified, this value will be used. Otherwise, the value in /etc/machine-id will be used. If this file is empty or missing, systemd will attempt to use the D-Bus machine ID from /var/lib/dbus/machine-id, the value of the kernel command line option container_uuid, the KVM DMI product_uuid or the devicetree vm,uuid (on KVM systems), and finally a randomly generated UUID.

In respect to other init systems, I didn't find documentation about how the machine-id is generated. But I would assume they do the right thing.

Taking into account that and the lack of activity of this ticket during 4+ years, I don't see any actionable thing to do from a cloud-init's perspective, so maybe this ticket should be closed.

This discussion was restarted by https://github.com/canonical/cloud-init/pull/1737.

[1] https://www.freedesktop.org/software/systemd/man/machine-id.html#Initialization

[Chad Smith (chad.smith)]

Agreed Alberto. I believe we've determined from cloud-init upstream that cloud-init specifically isn't the service to set and update machine-id on cloned golden images. It is generally up to the image creator to unset that field and the underlying init system to ensure that is properly configured in the case of cloned VMs.

That said have added an upstream commmit[1] to allow golden image creators to run:
  sudo cloud-init clean --logs --machine-id

This command will be available in cloud-init 22.3 and later will perform that /etc/machine-id cleanup and is our desired best practice for folks creating golden images in clouds.

[1] https://github.com/canonical/cloud-init/commit/8f569d45111dce368f92883e35bfbb40bbb86091

[Chad Smith (chad.smith)]

Marking Won't Fix as our status here. If there are compelling use-cases we need to reconsider. Please feel free to reopen this bug.

[[e-mail] (eblock)]

I'd like to propose a change regarding the machine-id setup, but I'm not sure how to reopen this bug. If you just remove the file /etc/machine-id (for example via 'cloud-init clean --machine-id') it won't regenerated during startup but error messages are displayed (here in Ubuntu 20.04):

---snip---
Jan 9 14:49:48 control-4 kernel: [ 3.482774] systemd[1]: System cannot boot: Missing /etc/machine-id and /etc is mounted read-only.
Jan 9 14:49:48 control-4 kernel: [ 3.484066] systemd[1]: Booting up is supported only when:
Jan 9 14:49:48 control-4 kernel: [ 3.485226] systemd[1]: 1) /etc/machine-id exists and is populated.
Jan 9 14:49:48 control-4 kernel: [ 3.485861] systemd[1]: 2) /etc/machine-id exists and is empty.
Jan 9 14:49:48 control-4 kernel: [ 3.486569] systemd[1]: 3) /etc/machine-id is missing and /etc is writable.
Jan 9 14:49:48 control-4 kernel: [ 4.247850] systemd[1]: Failed to populate /etc with preset unit settings, ignoring: Read-only file system
---snip---

If you just truncate the files to size 0 the boot works fine:

root@control-4:~# truncate -s0 /etc/machine-id /var/lib/dbus/machine-id

I haven't tested many scenarios but this change might help a bit when creating VM images.

[Chad Smith (chad.smith)]

Thanks @eblock for the additional comment here.
These behavior (to truncate /etc/machine-id on systemd) can be manually triggered by cloud-init clean --machine-id as of the following commit https://github.com/canonical/cloud-init/commit/abfdf1d83995cc20eb929d98485c321afb3a73b9

It should be in the next official release of cloud-init version 23.1 in a couple weeks

[Chad Smith (chad.smith)]

Additional bug tracking cloud-init clean --machine-id truncation of /etc/machine-id
https://bugs.launchpad.net/cloud-init/+bug/1999680