Error on cloud-init when kernel parameter "iscsi_auto" is set

[Victor Sartori]

On server enlistment, when the kernel parameter "iscsi_auto" is set and the VM has a local disk (I don't know if the local disk is part of this error), an error is raised at the cloud-config level.
I have attached a screenshot of the error. If necessary, I have a video of the full boot process.

Another important thing: this environment is running on top of Cisco UCS.

I am available to assist in resolving the issue. If deemed necessary, we can schedule a call and conduct tests in a testing environment (where this error is occurring).

[Anton Troyanov (troyanov)]

Hello Victor!

Can you please share all the kernel parameters that are passed to the bootloader?

I think you should be able to fetch them from http://$MAAS_IP:5248/grub/grub.cfg-d0:94:66:33:8f:7f (note that MAC address should be changed)

[Victor Sartori (victor-sartori)]

Hi Anton!

Follow:

curl http://127.0.0.1:5248/grub/grub.cfg-00:25:b5:00:00:02
set default="0"
set timeout=0

menuentry 'Ephemeral' {
    echo 'Booting under MAAS direction...'
    linux /images/ubuntu/amd64/ga-24.04/noble/stable/boot-kernel nomodeset ro root=squash:http://127.0.0.1:5248/images/ubuntu/amd64/ga-24.04/noble/stable/squashfs ip=::::comic-boar:BOOTIF ip6=off overlayroot=tmpfs overlayroot_cfgdisk=disabled cc:\{'datasource_list': ['MAAS']\}end_cc cloud-config-url=http://127.0.0.1:5248/MAAS/metadata/latest/by-id/7f3bgp/?op=get_preseed log_host=127.0.0.1 log_port=5247 --- iscsi_auto BOOTIF=01-${net_default_mac}
    initrd /images/ubuntu/amd64/ga-24.04/noble/stable/boot-initrd
}

[Victor Sartori (victor-sartori)]

Anton, any news about this issue?

[Jacopo Rota (r00ta)]

@victor: IIRC you are using 24.04 as commissioning image. Does it work it 22.04?

[Anton Troyanov (troyanov)]

Victor, may I also ask you for the cloud-init config returned for this machine?
http://$MAAS_IP:5248/MAAS/metadata/latest/by-id/$SYSTEM_ID/?op=get_preseed

cloud-init errors out because somehow it is getting `static` PROTO (currently I have no idea where it is coming from)
https://github.com/canonical/cloud-init/blob/23136e6a94821320a85117a2e4c4bb9b0926541f/cloudinit/net/cmdline.py#L130

Maybe this bug is related https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2056187

[Victor Sartori (victor-sartori)]

@r00ta
> @victor: IIRC you are using 24.04 as commissioning image. Does it work it 22.04?

No, because we are affected by a bug in initramfs (https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2056187).

Using ubuntu 24.04, it seems we are not affected by this bug in initramfs.

@troyanov
folow the cloud-config

curl http://10.107.72.10:5248/MAAS/metadata/latest/by-id/7f3bgp/?op=get_preseed
#cloud-config
apt:
  preserve_sources_list: false
  sources_list: 'deb http://archive.ubuntu.com/ubuntu/ $RELEASE universe restricted
    multiverse main

    # deb-src http://archive.ubuntu.com/ubuntu/ $RELEASE universe restricted multiverse
    main

    deb http://archive.ubuntu.com/ubuntu/ $RELEASE-updates universe restricted multiverse
    main

    # deb-src http://archive.ubuntu.com/ubuntu/ $RELEASE-updates universe restricted
    multiverse main

    deb http://archive.ubuntu.com/ubuntu/ $RELEASE-security universe restricted multiverse
    main

    # deb-src http://archive.ubuntu.com/ubuntu/ $RELEASE-security universe restricted
    multiverse main

    deb http://archive.ubuntu.com/ubuntu/ $RELEASE-backports universe restricted multiverse
    main

    # deb-src http://archive.ubuntu.com/ubuntu/ $RELEASE-backports universe restricted
    multiverse main

    '
datasource:
  MAAS:
    consumer_key: na4jA3wmnw4Fhw7TKK
    metadata_url: http://10.107.72.10:5248/MAAS/metadata/
    token_key: cA4AtRZWv9CvtzcqBd
    token_secret: L2p4G6Q3WH4sAZMVhwv8ZFtJCQ7BAX4c
manage_etc_hosts: true
power_state:
  condition: test ! -e /tmp/block-poweroff
  delay: now
  mode: poweroff
  timeout: 3600
reporting:
  maas:
    consumer_key: na4jA3wmnw4Fhw7TKK
    endpoint: http://10.107.72.10:5248/MAAS/metadata/status/7f3bgp
    token_key: cA4AtRZWv9CvtzcqBd
    token_secret: L2p4G6Q3WH4sAZMVhwv8ZFtJCQ7BAX4c
    type: webhook
rsyslog:
  remotes:
    maas: 10.107.72.10:5247

To try to help with this issue, I'm attaching a video of the boot process of a server enlisted by MAAS using Ubuntu 24.04.
The screen recorder was stopped when the error appeared, but the boot process continued, and we can see there is no network on the server. The apt command fails.

[Victor Sartori (victor-sartori)]

Guys, thinking about the issue, I guess I found why "proto" is static.

Why I guess: Looking at Ubuntu 22.04, the file created at /run/net-(iscsi interface) has the variable PROTO defined as "static" (see the file content attached).

When we set iscsi_auto in the kernel parameter, the kernel configures (in my case) two network cards with static IPs (the iSCSI network). These IPs come from the Cisco UCS iSCSI interface configuration (I have to confess, I don't know how this works in depth). Probably, the script is looking at /run/net-(interface) and there we find the static IP configuration.

The problem with initramfs, in my opinion, is not the root cause. The scripts in initramfs are working properly (assigning IPs to iSCSI interfaces and assigning IPs via DHCP).

However, I could not stop the initramfs initialization after all the scripts run to see what is happening. If you have any ideas on how to do this, please let me know.

[Alexsander de Souza (alexsander-souza)]

the '/run/net-*' files should follow klibc's ipconfig format [1], but the documentation is misleading. In one place it says that PROTO should be one of 'dhcp', 'bootp', 'rarp' or 'none', but later it states that 'off', 'static' and 'none' means no autoconfiguration. cloud-init follows the former specification, while klibc implementation and open-iscsi scripts use the latter.

This can be fixed by:

A) fixing cloud-init (`cloudinit/net/cmdline.py:94`) to allow `off` and `static`

B) fixing open-iscsi (`debian/extra/initramfs/local-top/iscsi:172`) to use `none` instead of `static`

Either way, this is not a MAAS bug

[1] https://git.kernel.org/pub/scm/libs/klibc/klibc.git/plain/usr/kinit/ipconfig/README.ipconfig

[James Falcon (falcojr)]

See https://github.com/canonical/cloud-init/pull/5437 for the cloud-init fix.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-init - 24.2-0ubuntu1

---------------
cloud-init (24.2-0ubuntu1) oracular; urgency=medium

  * Upstream snapshot based on 24.2.
    List of changes from upstream can be found at
    https://raw.githubusercontent.com/canonical/cloud-init/24.2/ChangeLog
    - Bugs fixed in this snapshot: (LP: #2065787)

 -- Chad Smith <email address hidden> Tue, 02 Jul 2024 19:59:23 -0600