Jammy's cloud-init doesn't work on lxd 4.0

[Wouter van Bommel]

When trying to boot a Jammy container on a host that uses lxd/4.0 the cloud-init there fails to work properly.

The result is, that a ssh key for authentication isn't inserted in the host, denying ssh access.

Upgrading to lxd/5.0 does works around this issue, which provides a newer interface, but isn't a solution as lxd/4.0 is an LTS of only a few years old.

For a complete log see: https://pastebin.canonical.com/p/X9GN5GH8wv/

To reproduce, install the lxd 4.0/stable snap on a machine, configure it to inject a ssh key, and try to launch an jammy container there. One won't be able to ssh into the container.

An extensive thread about this can be found at: https://chat.canonical.com/canonical/pl/c1erobjybt8jzbgxguowxrddrc

[Alberto Contreras (aciba)]

Reproduced with:

```sh
lxc launch ubuntu-daily:focal fvm --vm
lxc shell fvm

snap list lxd
lxd init --minimal
lxc launch ubuntu-daily:jammy j
lxc exec j -- cloud-init status --wait

$ lxc exec j -- grep -i warning /var/log/cloud-init.log
2023-01-10 10:29:12,936 - util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceLXD.DataSourceLXD'> failed
2023-01-10 10:29:14,169 - activators.py[WARNING]: Running ['netplan', 'apply'] resulted in stderr output: Failed to connect system bus: No such file or directory
2023-01-10 10:29:27,476 - cc_final_message.py[WARNING]: Used fallback datasource

$ lxc exec j -- grep -i exception /var/log/cloud-init.log
     raise sources.InvalidMetaDataException(
cloudinit.sources.InvalidMetaDataException: Invalid HTTP response [404] from http://lxd/1.0/devices: 404 page not found
```

[Alberto Contreras (aciba)]

The problem is that cloud-init expects `/1.0/devices` to be available in the lxc instance socket, but this information was introduced in LXD 4.21, see [1].

This issue is present in focal (lxd 4.0/stable) and bionic (lxd 3 deb package).

As a workaround for focal images, one could perform an upgrade to 4.21:

sudo snap refresh --channel=4.21/stable lxd

A solution for cloud-init could be to only fetch /1.0/devices on compatible lxd versions and enable/disable the lxd hotplug feature depending on that.

[1] https://discuss.linuxcontainers.org/t/lxd-4-21-has-been-released/12860

[Chad Smith (chad.smith)]

cloudinit.sources.InvalidMetaDataException: Invalid HTTP response [404] from http://lxd/1.0/devices: 404 page not found

Cloud-init can better handle this error in the event that the LXD control plane doesn't fully support the devices endpoint.

Expectation is LXD 4.0 will not get this feature backported, and cloud-init has fallback configuration options when devices don't exist.

[Chad Smith (chad.smith)]

work in progress pull request which will resolve this problem. Needs unit tests and then cloud-init can support jammy launches from LXD 4.0 hosts.
https://github.com/canonical/cloud-init/pull/1948

[Chad Smith (chad.smith)]

This fix will be in the next SRU release of cloud-init version 23.1 (mid-february). It will be in the next interim upload to Lunar (23.04) as version 22.4.2-0ubuntu3

[Alberto Contreras (aciba)]

This bug is believed to be fixed in cloud-init in version 23.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

[James Falcon (falcojr)]

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/4062