cloud-init + installer not creating user accounts

[Hadmut Danisch]

Hi,

I'm currently testing the 20.04 methods of both automatically running and installing ubuntu server using cloud-init files and the autoinstaller (using methods like PXE, the provided image and the autoinstaller).

Unfortunately the documentation covers only the main parameters but does not tell the details of the autoinstallation process. I ran into the problem that while running a 20.04 server images with cloud-init files seems to create these users, while running the ubuntu autoinstaller (did not even find where it comes from) seems to mostly ignore account parameters.

In order to track down the problem I've stripped things down to a test case, see the attached install.sh and user-data files.

I've included three different accounts at different locations of the user-data file.

- hadmut1 in autoinstall:user-data: which is the only account information even arriving on the installed machine, contained in /var/log/installer/autoinstall-user-data , but not created. However, it is not completely ignored, since if I am ommitting the ssh key, there is a warning message when booting the machine about a missing ssh key.

- hadmut2 in autoinstall: seems to be completely irrelevant.

- hadmut3 at toplevel seems to be installed, but on the temporary machine running the autoinstaller only, since on the installed machine there's a /var/log/installer/installer-journal.txt telling that it had been created on the installer machine (nice, but useless).

So while machines run directly with cloud-init seem to create the user accounts in the users: section of user-data, the autoinstaller does not seem to do this, almost completely ignoring those users (except for noticing if the ssh pub key is missing).

Actually, I am not even sure whether the autoinstaller is even intended to create users or just the one given in the „identity” section. I did not even find any documentation about this. There is only a short paragraph in

https://ubuntu.com/server/docs/install/autoinstall-reference#user-data
and
https://ubuntu.com/server/docs/install/autoinstall-reference#identity

telling that this is mixed with the users section and at least one of them must be present, identity can be omitted if there's a user section, but that's not what's happening. Without the identity section, it is not possible to login at all.

So the autoinstaller seems to both differ from the documentation and to create machines where you can't login (while the documentation says that can't happen).

regards

[Hadmut Danisch (hadmut)]

Attached install.sh to demonstrate the problem (will download huge iso if not present)

[Paride Legovini (paride)]

Hello Hadmut and thanks for this bug report. From reading the information here I think user hadmut1 should be created, while I don't expect hadmut2 and hadmut3 to show up in the installed system. Specifying the user in the 'identity' section shouldn't be necessary when specifying users like you did for hadmut1.

I didn't have time to setup a reproducer today, I'll try next Monday. I doubt this is a cloud-init bug, however I'm fine with starting the investigation here.

[Hadmut Danisch (hadmut)]

I've made some progress with debugging.

- the account hadmut1 is created if the identity:-section is completely removed from user-data, so identity is not 'optional', but exclusive.

- it does not exist immediately when the first boot prompt comes, it is delayed. You have to wait until you see around one and a half screens full of log messages

- I could log in only over ssh and ssh-key with additional kvm option -nic user,model=virtio,hostfwd=tcp::2222-:22, not with password, because I had copied the password entry from the identity section to use the default 'ubuntu' password. Doesn't work, because in the identity:-section the password entry tag is password:, while in the users section it is passwd: . If you name it password:, it is not found and thus not set.

- even if the password is given with the passwd: tag, it is not possible to login with that password, because the entry in /etc/shadow looks like this:

hadmut1:!$6$exDY1mhS4KUYCE/2$zmn9ToZwTKLhCw.b4/b.ZRTIZM30JZ4QrOQ2aOXJ8yk96xpcCof0kxKwuX1kqLG/ygbJ1f8wxED22bTL4F46P0:18728:0:99999:7:::

Note the ! the hash begins with. That's inhibiting password use. However, the account is not completely locked, since login with ssh is still possible.

I found by googling, that there is an additional field

lock_passwd: true

which might have a default value of true. I'll check this.

[Hadmut Danisch (hadmut)]

Yes, a
lock_passwd: false

is required to enable the password.

So
- remove the identity section complety
- change password: key to passwd:
- add lock_passwd: false
- wait for some time after first login prompt

is what is needed to get the hadmut1 account and login with password 'ubuntu'. hadmut2 is never created and hadmut3 is temporarily created on the temporary machine performing the installation procedure.

So it's rather a problem of insufficient documentation than of cloud-init itself.

However, cloud-init had formerly already caused me plenty of headache and debugging work since it is not obvious that cloud-init remains active and changing the machine after first boot. Problems I had was that cloud-init resets a change hostname to what it has stored somewhere in a file, thus making it impossible to change the host name permanently, or recreating the ssh host keys because of timing problems when renaming the ethernet device twice, having problems to identify and believing to be on a new machine. All of these effects seem to not have been documented.

[Paride Legovini (paride)]

Hi Hadmut and thanks for your additional debugging work here and for sharing your findings.

I agree this looks like a documentation issue: especially the fact that the 'identity' and 'user-data/users' sections are mutually exclusive and 'lock_passwd' need better documentation and possibly some examples. I think this belongs to subiquity (the Ubuntu Server installer) and not to cloud-init, so I added a new subiquity bug task.

The fact the user does not exist immediately after rebooting the newly installed system is normal: cloud-init needs some time to finish initializing the system. This is done at first boot.

On the issue you mentioned about cloud-init remaining active after first boot: I encourage you to file bugs for specific issues, but it's very likely that the cloud-init behaviors that caused you headaches are due to design choices made to accommodate common cloud usage scenarios. This said: there is certainly room for improvement in many aspects and good bug reports are always more than welcome.

Back to the auto-install issue this report is about: I'm setting the cloud-init task to Invalid, and leave the subiquity task open. Should you still think this is a cloud-init bug please comment back and change the task status back to New, we'll look at this again. Thanks!