cloud-init runs too late at first startup after ubuntu autoinstall

Bug #1907107 reported by eoli3n
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
subiquity
Invalid
Undecided
Unassigned
cloud-init (Ubuntu)
Expired
Undecided
Unassigned

Bug Description

#cloud-config
autoinstall:
  version: 1
  # Timezone
  user-data:
    timezone: Europe/Paris

After end of installation and reboot ->

root@focal-uefi:~# cat /etc/timezone
Etc/UTC

root@focal-uefi:~# grep -i timezone /var/log/cloud-init.log
2020-12-07 14:06:01,468 - stages.py[DEBUG]: Running module timezone (<module 'cloudinit.config.cc_timezone' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_timezone.py'>) with frequency once-per-instance
2020-12-07 14:06:01,469 - handlers.py[DEBUG]: start: modules-config/config-timezone: running config-timezone with frequency once-per-instance
2020-12-07 14:06:01,470 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/iid-datasource-none/sem/config_timezone - wb: [644] 23 bytes
2020-12-07 14:06:01,471 - helpers.py[DEBUG]: Running config-timezone using lock (<FileLock using file '/var/lib/cloud/instances/iid-datasource-none/sem/config_timezone'>)
2020-12-07 14:06:01,474 - util.py[DEBUG]: Writing to /etc/timezone - wb: [644] 13 bytes
2020-12-07 14:06:01,480 - handlers.py[DEBUG]: finish: modules-config/config-timezone: SUCCESS: config-timezone ran successfully

Revision history for this message
eoli3n (eoli3neoli3n) wrote :
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Please use

$ ubuntu-bug cloud-init

this will file a new bug report against cloud-init collecting the data required to debug cloud-init. At this point it is a cloud-init issue.

Changed in subiquity:
status: New → Invalid
Revision history for this message
eoli3n (eoli3neoli3n) wrote :

I found what happens but i can't explain:

if I set a different hostname with early command, then timezone is not correctly set after reboot:

early-commands:
  - hostname whatever

Related to https://bugs.launchpad.net/subiquity/+bug/1905932

Revision history for this message
eoli3n (eoli3neoli3n) wrote :

I didn't wait at first startup for scripts to run to finish configurations, i ran my test too early.
Changing hostname doesn't break timezone.

Revision history for this message
eoli3n (eoli3neoli3n) wrote :

It seems that cloud-init runs too late.
When starting host, i get login prompt on tty1 then cloud-init starts running overriding and messing up output.

When I use installer to install desktop, lightdm starts before cloud-init has run.
Then, lightdm is in english (no locale is generated yet), and without timezone, clock is not well configured.
If i wait 3 or 4 min, then restart lightdm, its well configured.

Related :
https://bugs.launchpad.net/subiquity/+bug/1906829 : autoinstall: set locale to fr_FR not working

summary: - autoinstall: timezone is ignored
+ cloud-init runs too late at first startup after ubuntu autoinstall
Revision history for this message
Ryan Harper (raharper) wrote :

I'm marking the cloud-init task invalid for now.

As mentioned in the other bug[1]; cloud-init is not running too late; rather scripts not provided to cloud-init are not waiting for cloud-init to complete its execution. If your scripts are using cloud-init status --wait before executing or systemd units are gated with After=cloud-init.target, then please re-open this task by marking it New again. When doing so please attach ubuntu-bug cloud-init output and a sample of your scripts and how they're added to the image so we can see when they are running.

I do believe that if you gate your scripts to run after cloud-init has completed there shouldn't be any issues.

1. https://bugs.launchpad.net/subiquity/+bug/1906829/comments/11

Changed in cloud-init (Ubuntu):
status: New → Invalid
Revision history for this message
eoli3n (eoli3neoli3n) wrote :

I use autoinstall to get my system installed in a specific state, then i need to boot that state.
I want my tools to use that state (lightdm for exemple), so if cloud-config needs to run some tasks it should do it before i can interact with my system.

Is that normal that cloud-config starts after tty1 login, overriding prompt, messing up with term ?
lightdm is not a script, is that normal that it starts before cloud-config ?
lightdm starting with wrong timezone and locale is a bug, and not a lightdm one.
Should I modify manually all units to wait for cloud-init.target ? I don't think so.

Then cloud-config should run ealier in the boot process, and at first boot, lightdm should start with right timezone and locale set.

Revision history for this message
eoli3n (eoli3neoli3n) wrote :

As I manually setup a unit which plays with Before=lightdm.service, maybe it breaks something, i will test and tell later.

Revision history for this message
eoli3n (eoli3neoli3n) wrote :

You can reproduce with that minimal user-data config

#cloud-config
autoinstall:
  version: 1
  packages:
    - lightdm
  locale: fr_FR.UTF-8
  user-data:
    timezone: Europe/Paris
  refresh-installer:
    update: yes
  identity:
    hostname: localhost
    username: first
    password: changeme

After first reboot, lightdm start without locale and with wrong timezone.
If i switch to another tty, then i can see cloud-config running, messing up tty login prompt.
Check screenshot attached, you can see at the top login prompt.

Changed in cloud-init (Ubuntu):
status: Invalid → New
Revision history for this message
eoli3n (eoli3neoli3n) wrote :

Actually, its gdm which runs on tty1, as i did not enabled lightdm... but it doesnt change the problem.

Revision history for this message
eoli3n (eoli3neoli3n) wrote :

That screenshot show time when those services are ready
- gdm
- cloud-init
- cloud-config

Revision history for this message
Ryan Harper (raharper) wrote :

Can you run and attach the output tarball on the target system where you see this fail:

cloud-init collect-logs

With respect to your use-case; cloud-init does not directly control what packages do at install time. Some package may require post-install configuration, or are not typically installed during the boot process. Just testing your config, installing lightdm brings in over 500 packages and took about 12 minutes to install; This is a bit of an uphill struggle; I can say that the Ubuntu Server installer isn't typically used to install Ubuntu Desktop; have you looked at just using the Ubuntu Desktop installer with OEM mode? https://help.ubuntu.com/community/Ubuntu_OEM_Installer_Overview

I suspect you'll need to employ some additional changes to the cloud-config to handle these edge cases; I don't think the respective packages, like gdm, would change.

I would suggest that you add a runcmd to restart gdm or other packages which might need to restart after the initial install; reload any installed systemd units with daemon-reload and then restart gdm. runcmds are executed after package install.

runcmd:
  - systemctl daemon-reload
  - systemctl gdm restart

> If i switch to another tty, then i can see cloud-config running, messing up tty login prompt

cloud-init is configured to write to the console by default; that's why you see output from cloud-init on your tty.

Revision history for this message
Ryan Harper (raharper) wrote :

Please move this back to New after you've attached the cloud-init collect-logs output tarball.

Changed in cloud-init (Ubuntu):
status: New → Incomplete
Revision history for this message
eoli3n (eoli3neoli3n) wrote :

> installing lightdm brings in over 500 packages and took about 12 minutes to install;*
It seems normal to me, it installs a big tree of dependencies because its the first "graphic" package.

> Some package may require post-install configuration.
I used that method with success for many years with a kickstart install script.
It installs the system then run ansible-playbook in a chroot to install all packages and configure everythings.

> I can say that the Ubuntu Server installer isn't typically used to install Ubuntu Desktop.
I use it because i can't find any netboot installer which supports autoinstall and EFI installation. I used legacy netboot (60MB) for my kickstart install but now i need to autoinstall an EFI system.

> have you looked at just using the Ubuntu Desktop installer with OEM mode? https://help.ubuntu.com/community/Ubuntu_OEM_Installer_Overview
I just want a way to automate installation from a pxe boot, with an "answer file", i did with kickstart in a "script" model, i want to do the same with autoinstall.

> I would suggest that you add a runcmd to restart gdm or other packages which might need to restart after the initial install;
Thanks for the tip, i try this as soon as possible.

> cloud-init is configured to write to the console by default; that's why you see output from cloud-init on your tty.
It seems to me a little bit buggy, don't you think ?
I think cloud-config.service should defaulty have a stronger dependency as Before=getty.target for exemple.

Why don't the installer configure everything during the install process ? Generating locales, write /etc/default/locale and set /etc/timezone are not hard tasks. Even if cloud-init embeed those configuration, you could tell the installer to setup what in cloud-config as default config.

If you can't avoid that first startup scripts, you should start as early as possible in the init process, before any process that could use what it setup/modify. Does that seem illogical ?

Revision history for this message
eoli3n (eoli3neoli3n) wrote :

This is log collected after first boot with previous minimal user-data file (which only set locales, timezone and install lightdm package), i waited for cloud-config to run.

Changed in cloud-init (Ubuntu):
status: Incomplete → New
Revision history for this message
Ryan Harper (raharper) wrote : Re: [Bug 1907107] Re: cloud-init runs too late at first startup after ubuntu autoinstall

* eoli3n <email address hidden> [2020-12-08 11:50]:

> > I can say that the Ubuntu Server installer isn't typically used to install Ubuntu Desktop.
> I use it because i can't find any netboot installer which supports
> autoinstall and EFI installation. I used legacy netboot (60MB) for my
> kickstart install but now i need to autoinstall an EFI system.

OK

>
> > have you looked at just using the Ubuntu Desktop installer with OEM
> > mode?
> > https://help.ubuntu.com/community/Ubuntu_OEM_Installer_Overview
> I just want a way to automate installation from a pxe boot, with an
> "answer file", i did with kickstart in a "script" model, i want to do
> the same with autoinstall.

Sure

> > cloud-init is configured to write to the console by default; that's
> > why you see output from cloud-init on your tty.
> It seems to me a little bit buggy, don't you think ?

Well, that's not a bug. It's the default configuration for cloud-init
as it runs in places where there's no access to tty or console; just the
output that a cloud platform may show you.

You may want to remove/override the StandardOutput setting in the
cloud-init service units (cloud-init-local, cloud-init, cloud-config,
cloud-final). The default is to log to both journal and console.

# Output needs to appear in instance console output
StandardOutput=journal+console

It's something that subiquity could configure differently since in
this scenario, the target system would have user-facing console.

> I think cloud-config.service should defaulty have a stronger
> dependency as Before=getty.target for exemple.

Possibly. If you want to test that out, you can add this as a drop-in
conf, before for boot the target OS you can:

mkdir -p /target/lib/systemd/system/cloud-config.service.d
echo -e "[Unit]\nBefore=getty.target > /target/lib/systemd/cloud-config.service.d/10-before-getty.conf

>
> Why don't the installer configure everything during the install process
> ? Generating locales, write /etc/default/locale and set /etc/timezone
> are not hard tasks. Even if cloud-init embeed those configuration, you
> could tell the installer to setup what in cloud-config as default
> config.

The design is based on the idea of a template image. The Ubuntu server
image is ready to boot and on first boot cloud-init handles the tasks
to "instantiate the image".

Doesn't the autoinstall input generatea cloud-init config that
configures these things?

>
> If you can't avoid that first startup scripts, you should start as early
> as possible in the init process, before any process that could use what
> it setup/modify. Does that seem illogical ?

It's not. Cloud-init does start up as early as possible (after local fs
is mounted). However some tasks require to be run later; for example
package install cannot run without networking being configured.

At this point, I don't think there is a bug in cloud-init. For your
use-case, you may need change some of the defaults that are already
cofigured in the image.

It's probably worth looking at your working solution to see what, if
anything, could be configured by subiquity; or if it makes sense to
tweak a default.

Revision history for this message
eoli3n (eoli3neoli3n) wrote :

If i run autoinstall with that user-data file, autoinstall doesn't run and subiquity prompt for language setting.

#cloud-config
runcmd:
  - systemctl daemon-reload
  - systemctl restart gdm3
autoinstall:
  version: 1
  packages:
    - gdm3
  locale: fr_FR.UTF-8
  user-data:
    timezone: Europe/Paris
  refresh-installer:
    update: yes
  identity:
    hostname: localhost
    username: premier
    password: $1$l03JsARH$DCxHdrzUH8SRmeDk3/3rU1

If I use that user-data file, runcmd seems not to run at the end of cloud-config after first reboot.

#cloud-config
autoinstall:
  version: 1
  packages:
    - gdm3
  locale: fr_FR.UTF-8
  user-data:
    timezone: Europe/Paris
  refresh-installer:
    update: yes
  identity:
    hostname: localhost
    username: premier
    password: $1$l03JsARH$DCxHdrzUH8SRmeDk3/3rU1
  runcmd:
    - systemctl daemon-reload
    - systemctl restart gdm3

How should i use runcmd ?

FYI, even if i restart gmd3 manually with Fr locale set and generated, it is still in english.
Then only timezone test is ok, if i restart manually after cloud-config, clock is well configured.

Revision history for this message
eoli3n (eoli3neoli3n) wrote :

>The design is based on the idea of a template image. The Ubuntu server
image is ready to boot and on first boot cloud-init handles the tasks
to "instantiate the image".
>Doesn't the autoinstall input generatea cloud-init config that
configures these things?

As i understand it, autoinstall generate both cloud-init config AND template installed on the disk.
When you use cloud-init in main way, you use a default templated OS. Autoinstall replace that part and generate template on the disk, then it could define how the template is.

> mkdir -p /target/lib/systemd/system/cloud-config.service.d
>echo -e "[Unit]\nBefore=getty.target > /target/lib/systemd/cloud-config.service.d/10-before-getty.conf

Thanks, lets play with this and add a Before=gdm3.service.

Revision history for this message
Ryan Harper (raharper) wrote :

* eoli3n <email address hidden> [2020-12-08 13:30]:
> If i run autoinstall with that user-data file, autoinstall doesn't run
> and subiquity prompt for language setting.
>
> #cloud-config
> runcmd:
> - systemctl daemon-reload
> - systemctl restart gdm3
> autoinstall:
> version: 1
> packages:
> - gdm3
> locale: fr_FR.UTF-8
> user-data:
> timezone: Europe/Paris
> refresh-installer:
> update: yes
> identity:
> hostname: localhost
> username: premier
> password: $1$l03JsARH$DCxHdrzUH8SRmeDk3/3rU1
>
> If I use that user-data file, runcmd seems not to run at the end of
> cloud-config after first reboot.
>
> #cloud-config
> autoinstall:
> version: 1
> packages:
> - gdm3
> locale: fr_FR.UTF-8
> user-data:
> timezone: Europe/Paris
> refresh-installer:
> update: yes
> identity:
> hostname: localhost
> username: premier
> password: $1$l03JsARH$DCxHdrzUH8SRmeDk3/3rU1
> runcmd:
> - systemctl daemon-reload
> - systemctl restart gdm3
>
> How should i use runcmd ?

I think part of this confusion is that subiquity has it's autoinstall
format[1], which looks rather similar to cloud-config, but it's the same
and depending on *where* you need things to run (say in the installer
environment, or on firstboot) then you need to adjust the
config structure.

I believe if you want to pass a user-data section in the autoinstall
file[2].

It would look like this:

#cloud-config
autoinstall:
  version: 1
  locale: fr_FR.UTF-8
  refresh-installer:
    update: yes
  user-data:
    locale: fr_FR.UTF-8
    hostname: localhost
    users:
      - name: premier
        gecos: premier
        passwd: "$1$l03JsARH$DCxHdrzUH8SRmeDk3/3rU1"
        shell: /bin/bash
        groups: "admin,sudo"
        lock_passwd: False
    packages:
      - gdm3
    timezone: Europe/Paris
    runcmd:
      - systemctl daemon-reload
      - systemctl restart gdm3

The autoinstaller will merge the 'user-data' section of the autoinstall
cloud-config it normally generates and then place this config in the
target OS so that in first boot cloud init will do all of the things you
need.

>
> FYI, even if i restart gmd3 manually with Fr locale set and generated,
> it is still in english. Then only timezone test is ok, if i
> restart manually after cloud-config, clock is well configured.

Looking at the config I think if you use the user-data key in
autoinstall conf you can be sure that all of the required bits (locale,
timezone, packages, etc) do what's needed.

1. https://ubuntu.com/server/docs/install/autoinstall-reference
2. https://github.com/CanonicalLtd/subiquity/blob/main/examples/autoinstall-user-data.yaml

Ryan

Revision history for this message
Dan Watkins (oddbloke) wrote :

Marking this as Incomplete as Ryan has provided next debugging steps. Please do move it back to New once you've responded!

Changed in cloud-init (Ubuntu):
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for cloud-init (Ubuntu) because there has been no activity for 60 days.]

Changed in cloud-init (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.