LXD image metadata: Update nocloud-net templates in compressed LXD image metadata for releases older than 22.04 Jammy

[Thomas Hipp]

LXD images of releases older than 22.04 (Jammy) don't support the `cloud-init.*` config keys. This is confusing to users[0] seeing that the 22.04 LXD image as well as our own cloud-init enabled images support both `user.*` and `cloud-init.*` keys.

Could this possibly be changed so that older releases also support both sets of config keys?

[0] https://discuss.linuxcontainers.org/t/lxd-first-class-cloud-init-support/12559/20

[Thomas Bechtold (toabctl)]

Looking at cloud-init versions (Jammy, Focal and Bionic all have 22.2), I think this should be possible.

[Thomas Bechtold (toabctl)]

See also https://bugs.launchpad.net/cloud-images/+bug/1958460 for the initial discussion

[Chad Smith (chad.smith)]

Thanks ThomasH for the request here.

 Generally, cloud-init tries to avoid behavior changes on stable releases to avoid breaking external tooling folks may have grown to drive a given image on a cloud platform. A change like this does represent a minor corner case of behavioral change for cloud-init on LXD.

It would mean a shift for new instance launches from cloud-init detecting "no-cloud" datasource to the "lxd" datasource. While the datasources have near functional parity, there are some differences that would be presented to LXD users:
  1. DatasourceLXD relies on LXD socket API, so if permissions are not set `security.devlxd=true` in profiles cloud-init will detect DataSourceNone and have completely broken configuration
  2. DatasourceLXD does react to instance-id changes in LXD metadata across reboot, so systems will re-create ssh HostKeys if the instance-id presented by LXD changes across boot due to other `lxc config set` changes

These changes should only affect "new" launches of new images as you'll be dropping the LXD /var/lib/cloud/seed/nocloud-related metadata templates from the images. Any previous launches of old images would have written out those template files when the instance was originally launched. So, there should be no change in behavior for existing launched LXD images.

We have made "change in behavior" exceptions in the past for other clouds requesting those changes in default behavior (Oracle requesting that we move stable releases from being detected as DataSourceOpenStack -> DataSourceOracle for their customers). Since LXD is a cloud platform unto itself, you should be able to override typical Ubuntu stable behavior to make your product easier to manage.

I will perform a couple of tests locally on Jammy/Bionic/Focal etc but I presume if we just avoid presenting the /var/lib/cloud/seed files in LXD templates[1] we should be "good to go" here.

References:
 - https://blackboxsw.github.io/2021-11-08-customize-lxd-metadata-templates/#option-2-redact-nocloud-seed-templates-from-lxd-metadatayaml-and-templates

[Chad Smith (chad.smith)]

Also, consumers launching stable release images from LXD would be able to continue using NoCloud datasource by creating these metadata templates themselves in profiles before launching instances if they needed to retain NoCloud datasource behavior instead of the new LXD datasource for one reason of another.

[Chad Smith (chad.smith)]

@ThomasBechtold.
 I've tested bionic, focal and jammy with the following procedure:

cat > user-data <<EOF
#cloud-config
ssh_import_id: [chad.smith]
EOF

for series in bionic focal jammy; do
   lxc init ubuntu-daily/$series lp-1988401-$series
   lxc config set lp-1988401-$series cloud-init.user-data="$(cat user-data)"
   lxc config template delete lp-1988401-$series cloud-init-meta.tpl
   lxc config template delete lp-1988401-$series cloud-init-network.tpl
   lxc config template delete lp-1988401-$series cloud-init-user.tpl
   lxc config template delete lp-1988401-$series cloud-init-vendor.tpl
   lxc start lp-1988401-$series
   sleep 5
    lxc exec lp-1988401-$series -- cloud-init status --wait --long
    lxc exec lp-1988401-$series -- cloud-init query userdata
    lxc exec lp-1988401-$series -- cloud-init grep Traceback /var/log/cloud-init.log
done

All tests come up with LXD datasource properly, no Tracebacks and proper userdata being passed through to the instance.

[Chad Smith (chad.smith)]

Strike the Jammy portion of that test as Jammy, I didn't test that as Jammy already dropped the templates and metadata:

csmith@uptown:~/src/cloud-init$ lxc config metadata show lp-1988401-j
architecture: x86_64
creation_date: 1662086498
expiry_date: 0
properties:
  architecture: x86_64
  description: Ubuntu 22.04 LTS server (20220902)
  os: ubuntu
  release: jammy
templates:
  /etc/hostname:
    when:
    - create
    - copy
    create_only: false
    template: hostname.tpl
    properties: {}

lxc config template list lp-1988401-j
+--------------+
| FILENAME |
+--------------+
| hostname.tpl |
+--------------+

[Chad Smith (chad.smith)]

+1 from me for the drop of cloud-init LXD metadata and templates. Please inform cloud-init team in #cloud-init channel on LiberaChat when this change is released as we will have to fix our daily integration tests[1] by adding focal and bionic to expected series which don't contain NoCloud template files.

https://github.com/canonical/cloud-init/blob/main/tests/integration_tests/datasources/test_lxd_discovery.py#L39-L41

[Thomas Bechtold (toabctl)]

This is done now. all daily or release images for >= Bionic where the serial is >= 20220913 should work.
Closing now. Please reopen if there are problems.

[John Chittum (jchittum)]

Regression has been found for hosts running LXD3.0 and LXD4.0

We're calling LXD3.0 out of scope, but LXD 4.0 stream is what is default for Bionic and Focal. As such we'll revert the change until upstream LXD 4.0 is fixed

https://github.com/lxc/lxd/issues/10951#issuecomment-1258691195

[John Chittum (jchittum)]

reviewing with cloud-init team and lxd team, it appears this has been a latent bug in the 22.04+ LXD containers as well. The issue being a Host LXD 4.0 and any container relying on the LXD cloud-init datasource, along with the specific GoLang symbols (%s, though other special symbols may also be affected). original bug report against LXD:

https://github.com/lxc/lxd/issues/10231

This changes was not backported to LXD 4.0 branch, which is the LTS branch for Focal and Bionic. This means on Ubuntu Focal (20.04) and Bionic (18.04) hosts running a Jammy (22.04) guest or forward can still hit this specific bug

However, CPC released the change to 22.04 guests 7 months ago. The original bug in LXD shows that it surfaced in LXD 5.0 and was fixed there. This means 22.04 Host + 22.04 Guest is fine.

For 22.04 guests on hosts running LXD < 5, we want to ensure this bug is surfaced when it comes up so that users providing custom metadata see the issue.

[James Falcon (falcojr)]

"We're calling LXD3.0 out of scope, but LXD 4.0 stream is what is default for Bionic and Focal. As such we'll revert the change until upstream LXD 4.0 is fixed"
3 is default for Bionic, correct? "We'll revert the change until upstream LXD 4.0 is fixed" should only apply to focal, but not Bionic. For Bionic we'll keep the change reverted forever?

[John Chittum (jchittum)]

Yes, you are correct. Bionic is running LXD 3.X as a deb. Sorry for the confusion

We'll need to make a decision about Guest <-> Host support matrix. Based on current testing and usage, it appears the default assumption has been matching Host and Guest OS is supported, with no assumed state of unmatched Host / Guest.

With Bionic moving to ESM sooner rather than later, we'll need to make a decision soon on if we want Bionic Guests to be able to utilize new Host features available in LXD 4.0+, while also introducing a specific bug to LXD 3.0 based hosts...

[John Chittum (jchittum)]

reversion of the LXD Datasource metadata has been released for serial 20221003

I've set this ticket up to target specific series. At this point Jammy and Kinetic are done. However, these images will have the same bug on LXD 4.X and 3.X hosts (thus Jammy and Kinetic images have a known bug possibility on Bionic and Focal)

I'm setting to Bionic Won't Fix. this is because LXD has stated they will not backport to 3.X. thus to maintain full compatibility without pushing a known bug, we will not backport this change to Bionic containers

Focal containers are marked as Triaged. We know the work, and that LXD is considering the backport to 4.x LTS series for a possible November release. At that point, we will re-evaluate Focal containers moving to the LXD datasource.

Final point -- We need to make sure we (Canonical and CPC) have a strong public record of support of cross-Ubuntu version containers, forward and backwards (Jammy Container on Bionic Host and vice versa).