cloudconfig not writing maas data source

[Chris Norman]

further background https://discourse.maas.io/t/debian10-fails-on-final-reboot/4486

I'm deploying a debian 10 buster image using MAAS. No errors are reported in MAAS or curtin install but on the final boot cloud-init reports `Failed to load metadata and userdata`. checking the config for the machine in maas shows all the cloudconfig: to connect to maas but the files with oauth credentials etc dont seem to have been copied to the target.

[James Falcon (falcojr)]

Looking at the logs, a few things stand out:
 * /etc/cloud/cloud.cfg is practically empty. This is usually baked into the image, so to me this suggests a problem happening before we get to cloud-init.
 * ds-identify.log shows it couldn't detect a datasource. This means this check returned false, but it should be returning true on MAAS (https://github.com/canonical/cloud-init/blob/master/tools/ds-identify#L784).
 * The cloud-init code could also not detect a datasource. This again seems to suggest a problem with the image or fabric itself, not cloud-init.

I'm going to set the state of this bug to Incomplete for now. If you can provide more information to show that cloud-init isn't doing what you expected, feel free to set the status back to New.

[Scott Moser (smoser)]

When I looked at this, the most obvious thing to me was that curtin did not write a file in /etc/cloud/cloud.cfg.d/ that configured the MAAS datasource.

A quick look at curtin didn't remind me how that file gets written in ubuntu.
 https://github.com/canonical/curtin/blob/204728a21d1b0a6733a4f229d759b9ab324d9258/curtin/commands/curthooks.py#L1464

The 'cloud-config' entries provided by MAAS (https://discourse.maas.io/t/debian10-fails-on-final-reboot/4486/5) should then get written into /etc/cloud/cloud.cfg.d/ . If they aren't written, then the installed system won't recognize it is configured for maas.

[Chris Norman (cbnorman)]

thanks for looking into this and agreed the root problem seems to be the files are not written by curtin. I dont see any errors in the curtin log, do you know why that handle_cloudconfig function would not be run?

[Ryan Harper (raharper)]

On Thu, May 6, 2021 at 5:15 AM Chris Norman <email address hidden>
wrote:

> thanks for looking into this and agreed the root problem seems to be the
> files are not written by curtin. I dont see any errors in the curtin
> log, do you know why that handle_cloudconfig function would not be run?
>

MAAS sends this information through the debconf_selections curtin config:

https://curtin.readthedocs.io/en/latest/topics/config.html#debconf-selections

And the Ubuntu cloud-init package has a postinst which parses these
values and writes them out:

https://github.com/canonical/cloud-init/blob/ubuntu/daily/devel/debian/cloud-init.postinst#L54

> --
> You received this bug notification because you are subscribed to cloud-
> init.
> https://bugs.launchpad.net/bugs/1927020
>
> Title:
> cloudconfig not writing maas data source
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/1927020/+subscriptions
>

[Lee Trager (ltrager)]

As per discourse it looks like the bug is in Curtin. /etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg is written then deleted by Curtin.

Applying debconf selections
Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpihv32ogj/target', 'debconf-set-selections'] with allowed return codes [0] (capture=True)
Running command ['unshare', '--fork', '--pid', '--', 'chroot', '/tmp/tmpihv32ogj/target', 'dpkg-query', '--list'] with allowed return codes [0] (capture=True)
unconfiguring cloud-init
cleaning cloud-init config from: ['/tmp/tmpihv32ogj/target/etc/cloud/cloud.cfg.d/90_dpkg_local_cloud_config.cfg', '/tmp/tmpihv32ogj/target/etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg', '/tmp/tmpihv32ogj/target/etc/cloud/cloud.cfg.d/90_dpkg.cfg']

[Ryan Harper (raharper)]

> As per discourse it looks like the bug is in Curtin.

No

The point of removing any files in /etc/cloud/cloud.cfg.d is that dpkg-reconfigure *WILL* write out whatever values it needs from the debconf_selections.

So the two questions are:

1) in the maas config sent to this image does it include debconf_selections (I assume yes)
2) does the cloud-init package in the target filesystem include the cloudinit.postinstall from upstream cloud-init which handles the maas preseed ?

[Ryan Harper (raharper)]

Let's confirm whether the debian buster cloud-init package in the image contains a postinstall like upstream cloud-init. If not, then that's the bug. If it's present, can we confirm the MAAS curtin config included the debconf_selections.

[Chris Norman (cbnorman)]

i believe the maas config is sent to the image. here is the output from the maas server:

```
#maas admin machine get-curtin-config ykfy3r
Success.
Machine-readable output follows:
apt:
  preserve_sources_list: false
  proxy: http://192.168.50.10:8000/
  sources_list: 'deb http://deb.debian.org/debian buster main

    deb-src http://deb.debian.org/debian buster main

    deb http://security.debian.org/debian-security buster/updates main

    deb-src http://security.debian.org/debian-security buster/updates main

    '
cloudconfig:
  maas-cloud-config:
    content: "#cloud-config\ndatasource:\n MAAS:\n consumer_key: X7HqXFtzruufCcfjnF\n\
      \ metadata_url: http://localhost:5240/MAAS/metadata/\n token_key: M7VUBVr5KQwgVH3QZZ\n\
      \ token_secret: 8tJ8wUJNp6HxKSEtMwA8YyYtFzhpkGHL\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_cloud_config.cfg
  maas-datasource:
    content: 'datasource_list: [ MAAS ]'
    path: /etc/cloud/cloud.cfg.d/90_maas_datasource.cfg
  maas-reporting:
    content: "#cloud-config\nreporting:\n maas:\n consumer_key: X7HqXFtzruufCcfjnF\n\
      \ endpoint: http://localhost:5240/MAAS/metadata/status/ykfy3r\n token_key:\
      \ M7VUBVr5KQwgVH3QZZ\n token_secret: 8tJ8wUJNp6HxKSEtMwA8YyYtFzhpkGHL\n \
      \ type: webhook\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_cloud_init_reporting.cfg
  maas-ubuntu-sso:
    content: "#cloud-config\nsnappy:\n email: <email address hidden>\n"
    path: /etc/cloud/cloud.cfg.d/90_maas_ubuntu_sso.cfg
debconf_selections:
  grub2: grub2 grub2/update_nvram boolean false
  maas: 'cloud-init cloud-init/datasources multiselect MAAS

    cloud-init cloud-init/maas-metadata-url string http://localhost:5240/MAAS/metadata/

    cloud-init cloud-init/maas-metadata-credentials string oauth_consumer_key=X7HqXFtzruufCcfjnF&oauth_token_key=M7VUBVr5KQwgVH3QZZ&oauth_token_secret=8tJ8wUJNp6HxKSEtMwA8YyYtFzhpkGHL

    cloud-init cloud-init/local-cloud-config string manage_etc_hosts: true\nmanual_cache_clean:
    true\nreporting:\n maas:\n consumer_key: X7HqXFtzruufCcfjnF\n endpoint:
    http://localhost:5240/MAAS/metadata/status/ykfy3r\n token_key: M7VUBVr5KQwgVH3QZZ\n token_secret:
    8tJ8wUJNp6HxKSEtMwA8YyYtFzhpkGHL\n type: webhook\n

    '
install:
  error_tarfile: /tmp/curtin-logs.tar
  log_file: /tmp/install.log
  post_files:
  - /tmp/install.log
  - /tmp/curtin-logs.tar
kernel:
  fallback-package: linux-image-amd64
  package: linux-image-amd64
late_commands:
  datasource:
  - curtin
  - in-target
  - --
  - sh
  - -c
  - 'echo "datasource_list: [ MAAS ]" > /etc/cloud/cloud.cfg.d/90_dpkg.cfg'
  maas:
  - wget
  - --no-proxy
  - http://localhost:5240/MAAS/metadata/latest/by-id/ykfy3r/
  - --post-data
  - op=netboot_off
  - -O
  - /dev/null
network:
  config:
  - id: ens33
    mac_address: 00:0c:29:d7:99:89
    mtu: 1500
    name: ens33
    subnets:
    - address: 192.168.50.34/24
      dns_nameservers:
      - 192.168.50.10
      - 8.8.8.8
      dns_search: &id001
      - maas
      type: static
    type: physical
  - address:
    - 192.168.50.10
    search: *id001
    type: nameserver
  version: 1
network_commands:
  builtin:
  - curtin
  - net-meta
  ...

[Chris Norman (cbnorman)]

here is from the finished deployment

root@debian:~# cloud-init --version
/usr/bin/cloud-init 20.2
root@debian:~# apt list cloud-init
Listing... Done
cloud-init/stable 20.2-2~deb10u2 all [upgradable from: 20.2-2~deb10u1]
N: There is 1 additional version. Please use the '-a' switch to see it

[Ryan Harper (raharper)]

@Chris

Thanks for the info. I see maas is sending the debconf selections I would expect.
For the postinst, you can view it on an installed system with cloud-init here:

/var/lib/dpkg/info/cloud-init.postinst

And if this node was deployed by maas, you can also query the debconf database to see what's set:

debconf-show cloud-init

[Chris Norman (cbnorman)]

both seem to exist on the deployed machine

buster@debian:~$ ls /var/lib/dpkg/info/cloud-init.postinst
/var/lib/dpkg/info/cloud-init.postinst
buster@debian:~$ debconf-show cloud-init
debconf: DbDriver "passwords" warning: could not open /var/cache/debconf/passwords.dat: Permission denied
* cloud-init/local-cloud-config: manage_etc_hosts: true
manual_cache_clean: true
reporting:
  maas:
    consumer_key: V2j8cUq5WsPfHJTZtn
    endpoint: http://192.168.50.10:5248/MAAS/metadata/status/ykfy3r
    token_key: xAHTmw5FGknNAx8Lev
    token_secret: HA2ybunRP54ewHENZZK7BsFJyHuAUGFF
    type: webhook

* cloud-init/maas-metadata-url: http://192.168.50.10:5248/MAAS/metadata/
* cloud-init/maas-metadata-credentials: oauth_consumer_key=V2j8cUq5WsPfHJTZtn&oauth_token_key=xAHTmw5FGknNAx8Lev&oauth_token_secret=HA2ybunRP54ewHENZZK7BsFJyHuAUGFF
* cloud-init/datasources: MAAS

[Ryan Harper (raharper)]

What happens when you do:

sudo dpkg-reconfigure cloud-init --frontend=noninteractive

do you see /etc/cloud/cloud.cfg.d/ populated?

if not you can modify /var/lib/dpkg/info/cloud-init.postinst shebang to include execution trace:

#!/bin/sh -x

Then re-run the command and capture the output. Maybe we can see why that's not rendering the maas cloud config files.

FWIW, I've extracted the debconf_selections into a file, populated debconf, then re-run dpkg-reconfigure cloud-init and see that it does write out these files in /etc/cloud/cloud.cfg.d/

https://paste.ubuntu.com/p/9yt8nyqBdR/

[Chris Norman (cbnorman)]

not sure this is helpful?

root@debian:~# dpkg-reconfigure cloud-init --frontend=noninteractive
root@debian:~# ls /etc/cloud/cloud.cfg.d/
00_debian.cfg 00-users.cfg 05_logging.cfg 50-curtin-networking.cfg curtin-preserve-sources.cfg README
root@debian:~# vi /var/lib/dpkg/info/cloud-init.postinst
# changed shebang to #!/bin/sh -x
root@debian:~# dpkg-reconfigure cloud-init --frontend=noninteractive
+ . /usr/share/debconf/confmodule
+ [ ! 1 ]
+ [ -z ]
+ exec
+ [ ]
+ exec
+ DEBCONF_REDIR=1
+ export DEBCONF_REDIR
+ set -f
+ db_capb escape
+ _db_cmd CAPB escape
+ _db_internal_IFS=

+ IFS=
+ printf %s\n CAPB escape
+ IFS=

+ read -r _db_internal_line
+ IFS=

+ RET=multiselect escape
+ return 0
+ [ configure = configure ]
+ [ -z 20.2-2~deb10u1 ]
+ [ configure = configure ]
+ deb-systemd-helper unmask cloud-config.service
+ deb-systemd-helper --quiet was-enabled cloud-config.service
+ deb-systemd-helper update-state cloud-config.service
+ [ configure = configure ]
+ deb-systemd-helper unmask cloud-final.service
+ deb-systemd-helper --quiet was-enabled cloud-final.service
+ deb-systemd-helper update-state cloud-final.service
+ [ configure = configure ]
+ deb-systemd-helper unmask cloud-init-local.service
+ deb-systemd-helper --quiet was-enabled cloud-init-local.service
+ deb-systemd-helper update-state cloud-init-local.service
+ [ configure = configure ]
+ deb-systemd-helper unmask cloud-init.service
+ deb-systemd-helper --quiet was-enabled cloud-init.service
+ deb-systemd-helper update-state cloud-init.service
+ which py3compile
+ py3compile -p cloud-init
+ which pypy3compile
+ [ configure = configure ]
+ [ -x /etc/init.d/cloud-init-local ]
+ update-rc.d cloud-init-local defaults
+ [ configure = configure ]
+ [ -x /etc/init.d/cloud-init ]
+ update-rc.d cloud-init defaults
+ [ configure = configure ]
+ [ -x /etc/init.d/cloud-config ]
+ update-rc.d cloud-config defaults
+ [ configure = configure ]
+ [ -x /etc/init.d/cloud-final ]
+ update-rc.d cloud-final defaults
root@debian:~# ls /etc/cloud/cloud.cfg.d/
00_debian.cfg 00-users.cfg 05_logging.cfg 50-curtin-networking.cfg curtin-preserve-sources.cfg README

[Ryan Harper (raharper)]

Yeah, that sure looks like the maas preseed code in postinst isn't in the debian package, can you compare your file with:

https://github.com/canonical/cloud-init/blob/ubuntu/daily/devel/debian/cloud-init.postinst#L54

specifically, does it have the handle_preseed_maas() function? I suspect not, in which case you'd need that in your cloud-init package in debian for the image to be deployable via maas.

Alternatively, if maas does not send the debconf_selections, then the clean-up and dpkg-reconfigure path won't be taken in curtin.

[Chris Norman (cbnorman)]

ok this is definetly the problem. the handle_preseed_maas() is there but a lot shorter so i just replaced the whole file and it worked perfectly. I use http://cdimage.debian.org/cdimage/openstack/archive/10.8.0/debian-10.8.0-openstack-amd64.raw to create the image. i had previously tried apt upgrade cloud-init and that didnt help, what package contains the /var/lib/dpkg/info/cloud-init.postinst file?

[Ryan Harper (raharper)]

> ok this is definetly the problem. the handle_preseed_maas() is there but a lot shorter so i just replaced the whole file and it worked perfectly.

\o/

> what package contains the /var/lib/dpkg/info/cloud-init.postinst file

cloud-init itself. It looks like you should file a bug against debian's cloud-init package and mention that it's not tracking what upstream cloud-init (upstream cloud-init uses release branches to keep their debian/ directory changes for packaging).

> I use http://cdimage.debian.org/cdimage/openstack/archive/10.8.0/debian-10.8.0-openstack-amd64.raw to create the image.

You could install cloud-init from Ubuntu, there's a daily PPA from which you could pull a deb

https://launchpad.net/~cloud-init-dev/+archive/ubuntu/daily

[Ryan Harper (raharper)]

I'm marking the curtin task invalid; after getting config and logs we confirmed that the cloud-init package in the Debian image does not contain a recent handle_maas_preseed() that Ubuntu uses and curtin/MAAS rely upon to install the MAAS datasource correctly.

[Ryan Harper (raharper)]

I'm marking the cloud-init task invalid. After getting config and logs we confirmed that the cloud-init package in the Debian image does not contain a recent handle_maas_preseed() that Ubuntu uses and curtin/MAAS rely upon to install the MAAS datasource correctly.

[Scott Moser (smoser)]

>> what package contains the /var/lib/dpkg/info/cloud-init.postinst file

> cloud-init itself. It looks like you should file a bug against debian's
> cloud-init package and mention that it's not tracking what upstream
> cloud-init (upstream cloud-init uses release branches to keep their
> debian/ directory changes for packaging).

That really doesn't seem like the right option to me. Using dpkg preseed
"should work", but that seems like an arcane mechanism for
providing configuration data. It was a solution that was put in place in
ubuntu very early on (probably 12.04) when there wasn't a better option.

There now *is* a better option: the cloud-config files.

The better path forward would be to change curtin to write the
cloud-config files itself as is done in Centos or Ubuntu core.

I suggest that the bug here is in curtin (curtin-hooks). They needed to be
aware of this and write the files.

[Ryan Harper (raharper)]

> I suggest that the bug here is in curtin (curtin-hooks). They needed to be
aware of this and write the files.

This is really orchestrated by maas...

MAAS can choose to send debconf_selections *OR* cloud-config; but it sends both.
It's been sending both for quite some time. I'd like to see MAAS decide to no longer send debconf_selections and curtin could write out cloudconfig if present in the default curtin-hooks. In fact the maas images created by maas-image-builder include a curtin-hook which already writes out cloudconfig if present.

The decision to not render cloudconfig from curtin's default curthook has always been controlled by MAAS. I don't recall exactly why but I suspect that MAAS might not decide what config to send on a per image basis; ie they send a superset of config which works for all images (old an new) and for a range of curtin releases which don't render cloud-config for ubuntu but expect debconf_selections to do so.

[Chris Norman (cbnorman)]

i believe that when you set
apt:
  preserve_sources_list: true
in the curtin_userdata then debconf is not sent by maas, this is where i initially started. I'm not sure how to confirm this though?

[Ryan Harper (raharper)]

On Fri, May 14, 2021 at 10:16 AM Chris Norman <email address hidden>
wrote:

> i believe that when you set
> apt:
> preserve_sources_list: true
> in the curtin_userdata then debconf is not sent by maas, this is where i
> initially started. I'm not sure how to confirm this though?
>

I don't think so, maybe some MAAS folks can confirm this. Looking at the
preseeds, it's in the template.

https://github.com/maas/maas/blob/31a95206afe563012c48a20bcd8492fa32c7f405/contrib/preseeds_v2/curtin_userdata

MAAS always sends debconf_selections for cloud-init as that originally was
the only way MAAS had to configure a cloud-init
datasource. Over time curtin added support for writing out cloudconfig
into the target image. For various reasons MAAS
hasn't stopped sending debconf_selections to configure cloud-init in the
target image and the Ubuntu cloud-init package has
always had support for setting up the MAAS datasource via that method.

https://github.com/maas/maas/blob/1aa6276c0b6d7d702f1ed3036fd11d30e879c285/src/maasserver/compose_preseed.py#L399

There's not much logic on versions or anything; I think it's just been part
of the template.

> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1927020
>
> Title:
> cloudconfig not writing maas data source
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/cloud-init/+bug/1927020/+subscriptions
>

[Albert Valiev (artscout)]

Just in case anyone looking for answer, I managed to find solution, not elegant one but it works:

I defined node_distribution earlier in code:

{{py:
node_distribution = node.get_distro_series()
}}

1. cloud-init should be already installed in the image used for custom install. Both buster/bullseye
2. in case of bullseye python2 should be installed and made to provide /usr/bin/python, also pyyaml should be installed for python2. I did that with this late commands:

{{if node_distribution == "bullseye"}}
  27_install_python2: ["curtin", "in-target", "--", "sh", "-c", "DEBIAN_FRONTEND=noninteractive apt-get -y install python2 python-is-python2"]
  28_install_python2_yaml: ["curtin", "in-target", "--", "sh", "-c", "wget https://bootstrap.pypa.io/pip/2.7/get-pip.py && python2 get-pip.py && python2 -m pip install pyyaml"]
{{endif}}

3. netplan.io should be installed, while it's not used MaaS expects one to be installed on host so we install it:

  34_add_neplan: ["curtin", "in-target", "--", "sh", "-c", "DEBIAN_FRONTEND=noninteractive apt-get -y install netplan.io"]

4. postinstall action for cloud-init package should be fired in late commands, no other way (like dpkg-reconfigure or new install of cloud-init works normally, latter adding maas datasource but deleting network config sadly). And as datasource for maas defined in debconf we have to fire it so handle_maas function do it's magic:

{{if node_distribution == "bullseye" or node_distribution == "buster"}}
  35_reconfigure_cloud-init: ["curtin", "in-target", "--", "sh", "-c", "/var/lib/dpkg/info/cloud-init.postinst configure || true"]
{{endif}}

with this steps debian buster and bullseye installed successfully with MaaS 3.2.4

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3873