Subiquity doesnt honor ip= kernel cmdline when configuring the network

[Fabio Augusto Miranda Martins]

When booting from the live ISO installer, one can opt to edit the kernel cmdline so as to provide the network configuration to be used during an automated install (instead of relying on the user-data configuration).

Back in debian-installer, you could use something like this in the boot line to pass the configuration:

hostname=<hostname> domain=<domain> interface=<NIC> netcfg/get_ipaddress=<IP>
netcfg/get_netmask=<netmask> netcfg/get_gateway=<gateway>

In subiquity, theoretically we should be able to use the ip= kernel option for this purpose, however this option is ignored and it still tries to use dhcp.

In the example below, I used the following cmdline to boot the live installer:

initrd=/casper/initrd autoinstall ip=192.168.123.171::192.168.123.1:255.255.255.0:client-hostname:off ds=nocloud-net;s=http://192.168.123.237/ ---

However, we still used DHCP to configure the NIC:

Sep 1 20:35:13 ubuntu-server systemd-networkd[1247]: enp1s0: Gained carrier
Sep 1 20:35:13 ubuntu-server systemd-networkd[1247]: enp1s0: DHCPv4 address 192.168.123.170/24 via 192.168.123.1

I've used the latest 22.04 server ISO in this test:
https://releases.ubuntu.com/22.04/

[Michael Hudson-Doyle (mwhudson)]

If possible could you attach the contents of /etc/netplan, /run/systemd/network and /run/net-*.cfg ?

[Fabio Augusto Miranda Martins (fabio.martins)]

There you go:

- /etc/netplan:

ubuntu-server@ubuntu-server:/etc/netplan$ ls -l
total 8
-rw-r----- 1 root adm 117 Sep 1 21:41 00-installer-config.yaml
-rw-r--r-- 1 root root 529 Sep 1 21:40 50-cloud-init.yaml.dist-subiquity
ubuntu-server@ubuntu-server:/etc/netplan$ cat 00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    enp1s0:
      dhcp4: true
  version: 2
ubuntu-server@ubuntu-server:/etc/netplan$ cat 50-cloud-init.yaml.dist-subiquity
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
    ethernets:
        zz-all-en:
            dhcp4: true
            match:
                name: en*
        zz-all-eth:
            dhcp4: true
            match:
                name: eth*
    version: 2

- /run/systemd/network/:

ubuntu-server@ubuntu-server:/etc/netplan$ cd /run/systemd/network/
ubuntu-server@ubuntu-server:/run/systemd/network$ ls -l
total 4
-rw-r--r-- 1 root root 102 Sep 1 21:41 10-netplan-enp1s0.network
ubuntu-server@ubuntu-server:/run/systemd/network$ cat 10-netplan-enp1s0.network
[Match]
Name=enp1s0

[Network]
DHCP=ipv4
LinkLocalAddressing=ipv6

[DHCP]
RouteMetric=100
UseMTU=true

- There's no /run/net-*.cfg:

ubuntu-server@ubuntu-server:/run/systemd/network$ ls -l /run/net-*.cfg
ls: cannot access '/run/net-*.cfg': No such file or directory
ubuntu-server@ubuntu-server:/run/systemd/network$ cat /run/net-*.cfg
cat: '/run/net-*.cfg': No such file or directory
ubuntu-server@ubuntu-server:/run/systemd/network$ ls -ld /run/net*
drwxr-xr-x 2 root root 40 Sep 1 21:41 /run/netns
drwxr-xr-x 2 root root 40 Sep 1 21:40 /run/netplan
drwxr-xr-x 2 root root 60 Sep 1 21:40 /run/network
ubuntu-server@ubuntu-server:/run/systemd/network$ cd /run/
ubuntu-server@ubuntu-server:/run$ sudo find . -name *.cfg
./cloud-init/cloud.cfg
ubuntu-server@ubuntu-server:/run$ cat ./cloud-init/cloud.cfg
datasource_list: [ NoCloud, None ]
ubuntu-server@ubuntu-server:/run$

[Michael Hudson-Doyle (mwhudson)]

Hmm I made a mistake and the files should be called /run/net-*.conf but they are not there either. Can you attach /var/log/cloud-init.log?

[Michael Hudson-Doyle (mwhudson)]

Also, were you running those commands at the end of an install or as the first page of the installer was showing? It would be very interesting to see the state of /etc/netplan and /run/net-*.conf before switching to the real rootfs, I think if you put break=bottom or break=casper-bottom on the kernel command line it will stop around the right point.

[Fabio Augusto Miranda Martins (fabio.martins)]

I'm setting those options as the first screen shows up. I'm hitting F6 (Other options) and then changing:

- From:
Boot Options initrd=/casper/initrd quiet ---

- To:
Boot Options initrd=/casper/initrd autoinstall ip=192.168.123.171::192.168.123.1:255.255.255.0:client-hostname:off ds=nocloud-net;s=http://192.168.123.237/ ---

And then the installation proceeds.

I collected the logs that I uploaded to this bug while the installation was still in progress (the target rootfs was mounted at /tmp). Do you still need the break=bottom or break=casper-bottom test?

ubuntu-server@ubuntu-server:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.9G 0 3.9G 0% /dev
tmpfs 796M 1.3M 795M 1% /run
/dev/sr0 1.3G 1.3G 0 100% /cdrom
/dev/loop0 383M 383M 0 100% /rofs
/cow 3.9G 22M 3.9G 1% /
/dev/loop2 60M 60M 0 100% /usr/lib/modules
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
tmpfs 3.9G 263M 3.7G 7% /tmp
/dev/loop3 62M 62M 0 100% /snap/core20/1328
/dev/loop4 44M 44M 0 100% /snap/snapd/14978
/dev/loop5 68M 68M 0 100% /snap/lxd/21835
/dev/loop6 32M 32M 0 100% /snap/subiquity/3119
tmpfs 796M 0 796M 0% /run/user/999
/dev/vda3 31G 49M 29G 1% /target
/dev/vda4 4.9G 20M 4.6G 1% /target/tmp

[Fabio Augusto Miranda Martins (fabio.martins)]

Also, I tried break=bottom or break=casper-bottom, but in both situations it breaks into a shell during initramfs. I did check /etc/netplan and it doesn't exist and I don't have /run/net-*.conf. I do have /run/netplan directory, but it is empty.

[Michael Hudson-Doyle (mwhudson)]

Hm ok there _definitely_ should be some /run/net-*.conf files when you break into the initramfs. I should probably try to recreate and debug this myself but one more thing: can you pass 'debug' on the kernel command line and then attach /run/initramfs/initramfs.debug from the booted system?

[Fabio Augusto Miranda Martins (fabio.martins)]

Uploaded the /run/initramfs/initramfs.debug file. I have a VM running in our Lab that I can give you access if you want to use it for debugging. Let me know (fabiomirmar on MM)

[Michael Hudson-Doyle (mwhudson)]

Ah hah:

+ ipconfig -t 2 -d 192.168.123.171::192.168.123.1:255.255.255.0:client-hostname:off
ipconfig: off: SIOCGIFINDEX: No such device
ipconfig: no devices to configure

I think you probably mean "192.168.123.171::192.168.123.1:255.255.255.0:all:none" as your ip= option?

I wonder if we should fail the boot in this situation.

[Fabio Augusto Miranda Martins (fabio.martins)]

Thank you very much for the pointer, Michael. I totally missed this.

So, this is the setting I ended up using, which worked pretty well:

initrd=/casper/initrd autoinstall ip=192.168.123.180::192.168.123.1:255.255.255.0:fabiomirmar:enp1s0:off:192.168.123.1:8.8.8.8: ds=nocloud-net;s=http://192.168.123.237/ ---

The valid syntax is:

ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>:
   <dns0-ip>:<dns1-ip>:<ntp0-ip>

Documented in:
https://www.kernel.org/doc/Documentation/filesystems/nfs/nfsroot.txt

With that, the correct netplan yaml was created:

ubuntu-server@ubuntu-server:~$ sudo cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    enp1s0:
      addresses:
      - 192.168.123.180/24
      gateway4: 192.168.123.1
      nameservers:
        addresses:
        - 192.168.123.1
        - 8.8.8.8
  version: 2

Post installation, the configuration remains correct:

root@localhost:~# cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
  ethernets:
    enp1s0:
      addresses:
      - 192.168.123.180/24
      gateway4: 192.168.123.1
      nameservers:
        addresses:
        - 192.168.123.1
        - 8.8.8.8
  version: 2

root@localhost:~# cat /etc/hostname
localhost

The hostname isn't preserved, but I believe I get get away by adding the hostname entry under Identity (or later through cloud-init user-data).

Thank you very much once again!

I'll add a separate comment to ask about how to track a suggestion of a feature request.

[Fabio Augusto Miranda Martins (fabio.martins)]

My feature request question is:

With this new ip= approach, all we can provide is:

ip=<client-ip>:<server-ip>:<gw-ip>:<netmask>:<hostname>:<device>:<autoconf>:<dns0-ip>:<dns1-ip>:<ntp0-ip>

So, we can't install through a VLAN interface and/or through a lacp bond.

Back in d-i, we can use netcfg/use_vlan and netcfg/vlan_id through the installer cmdline, so as to configure a VLAN tagged interface and install over it, but apparently we can't do the same with subiquity. It would be nice to have this as a feature request for the future, for cases where the client switches won't allow untagged network traffic.

Also, the same applies for lacp. Sometimes the network ports are configured with lacp bond (802.3ad) and the switch won't let an interface be brought up unless it sends lacpdus. That feature is also missing in d-i, so although it would be nice to have, people have been able to get workarounds up to date.

That feature was discussed in:

Debian Bug report logs - #611250
please support network bonding
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=611250

In the Debian bug report that, someone even submitted a patch (see Message #39 received at <email address hidden>):

https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=611250;filename=bonding_v2.patch;msg=39

However, I've looked into debian-installet netcfg tree and that patch has never been merged:

https://salsa.debian.org/installer-team/netcfg

It has also never landed into Ubuntu's netcfg package:

https://launchpad.net/ubuntu/+source/netcfg

So, it means that the netcfg/bonding, netcfg/bonding_mode and netcfg/bonding_slaves options that the patch adds, were never really implemented.

Not sure what your thoughts are, but sounds like good features to implement in the future.

[Michael Hudson-Doyle (mwhudson)]

The new installer is an _entirely_ different codebase than d-i. Whether d-i supports something has no bearing at all on whether the new installer does.

You can install through a vlan, using vlan=$name.$id:$parent syntax and then using the $name.$id name in the ip= stanza, you can see an example (for s390x) on this page https://ubuntu.com/server/docs/install/lpar-autoinstall-on-s390x:

ip=10.11.12.42::10.11.12.1:255.255.255.0:zlinlpar:encc000.4711:none:10.11.12.1 vlan=encc000.4711:encc000 url=...

There doesn't seem to be any support in our initramfs for LACP bonds, though. Could you file a separate bug (at https://bugs.launchpad.net/ubuntu/+source/initramfs-tools probably) about that?

[Fabio Augusto Miranda Martins (fabio.martins)]

Thank you for the vlan pointer, Michael.

I'm aware that the new installer is completely different. The d-i installer examples were provided just to explain what I was looking for. The vlan information helps.

I also filed a bug under initramfs-tools for the lacp support:

https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1989618

Thanks again.