IPv6 iBFT boot runs into a timeout

[Benjamin Drung]

[ Impact ]

This bug affects IPv6 only networks in combination with booting from iSCSI Boot Firmware Table (iBFT). In those setups, there is no DHCPv4 available. The initramfs will only try DHCPv4 and run into a timeout.

[ Test Plan ]

This setup was distilled down to test cases and the new qemu-net-iscsi autopkgtest was added.

The test plan is to check that all autopkgtest were successful. In addition testing this version in a cloud environment is possible.

[ Where problems could occur ]

The newly added code only increases the initrd by some bytes. The changed code is around the network configuration in the initrd and can influence network boot with iSCSI. Users with IPv6 in iBFT might still want to have IPv4 configured. That should be done by the network daemon of the installed system.

[ Original report ]

iSCSI Boot Firmware Table (iBFT) can provide the network configuration that is needed to boot over iSCSI. The iBFT can contain IPv4 or IPv6 addresses. local-top/iscsi calls `configure_networking`. The default behaviour of `configure_networking` is DHCPv4 on all available interfaces (if ip= and ip6= are not set). If iBFT provides IPv6 addresses and no DHCPv4 server are running, `configure_networking` will only try DHCPv4 and run into a timeout.

Placing the single line "ISCSI_AUTO=true" into /etc/iscsi/iscsi.initramfs, or use the kernel boot line option "iscsi_auto" will configure the network devices based on iBFT, but the code in `local-top/iscsi` will assume that the addresses are IPv4.

dracut on the other hand has the kernel boot line parameters `rd.iscsi.firmware=1` and `rd.iscsi.ibft` and `ibft` as option for the `ip` parameter. The `ibft` option is a bit under documented. https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/installation_guide/chap-anaconda-boot-options mentions `ip=ibft` and dracut.cmdline mentions it.

There are multiple ways to address the timeout with IPv6 iBFT:

1) Extend `local-top/iscsi` to differentiate between IPv4 and IPv6 when "iscsi_auto" is used. This information needs to be passed to `configure_networking` to only try to bring up that device and protocol.

2) Use `ip=ibft` as indicator to use ibft as source for the network configuration. If no ip= parameter is set, local-top/iscsi should default to ip=ibft when calling configure_networking.

Both solutions require passing data to configure_networking.

Requiring the user to set `ip=off ip6=on` is not a good user experience and not a solution in my opinion.

[Benjamin Drung (bdrung)]

After discussion with Julian, we came to the conclusion that I will try to implement ip=ibft in initramfs-tools.

The first step will be to write test cases for it. At least two:
* one ibft with IPv4 (two interfaces on the client: one with DHCP server on the other side and one without DHCP server)
* one ibft with IPv6

[Benjamin Drung (bdrung)]

Note that this commit deprecated ip=ibft in dracut: https://github.com/dracut-ng/dracut-ng/commit/5580e4c176c92624054691a8e12cffe8622cc1a0

[Benjamin Drung (bdrung)]

I pushed the proposed fix for initramfs-tools to https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/2091904

Besides adding autopkgtest, the relevant code change is in https://git.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/commit/?id=0d384c77a8f9ba4e63a0746b54f03cb754339c1f:

```
diff --git a/scripts/functions b/scripts/functions
index 1a0880db..96a5e0d9 100644
--- a/scripts/functions
+++ b/scripts/functions
@@ -290,10 +290,34 @@ _update_ip_param()
        fi
 }

+_set_bootif_and_ip_param_from_ibft()
+{
+ local iface mac
+ for iface in /sys/firmware/ibft/ethernet*; do
+ echo "iface: $iface"
+ [ -e "$iface/mac" ] || continue
+ read -r mac < "${iface}"/mac
+ [ -n "$mac" ] || continue
+ echo "mac: $mac"
+ BOOTIF="$mac"
+ case "$(cat "$iface/ip-addr")" in
+ *:*)
+ # IPv6 address
+ IP=off
+ IP6=on
+ ;;
+ esac
+ done
+}
+
 configure_networking()
 {
        local netdev_desc

+ if [ -z "${DEVICE-}${DEVICE6-}${BOOTIF-}" ] && [ -e /sys/firmware/ibft ]; then
+ _set_bootif_and_ip_param_from_ibft
+ fi
+
        # The order of precedence here is:
        # 1. Device specified by ip= kernel parameter
        # 2. Device matching MAC specified by BOOTIF= kernel parameter
```

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Benjamin Drung (bdrung)]

Pushed the proposed fix for plucky and noble to https://launchpad.net/~bdrung/+archive/ubuntu/initrd

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.145ubuntu2

---------------
initramfs-tools (0.145ubuntu2) plucky; urgency=medium

  * Fix shellcheck complaints in debian/tests
  * configure_networking: Configure IPv4 or IPv6 based on iBFT IP address
    (LP: #2091904)
  * test:
    - support setting a disk label in build_fs_ext2
    - test-common: move disc logic out of _run_qemu
    - move wait_for_server_startup to test-common
    - check for QEMU PID in wait_for_server_startup
    - Add qemu-net-iscsi autopkgtest for iBFT test cases

 -- Benjamin Drung <email address hidden> Thu, 06 Feb 2025 22:21:42 +0100

[Benjamin Drung (bdrung)]

Uploaded SRUs: initramfs-tools 0.142ubuntu34.2 for oracular and 142ubuntu25.5 for noble.

[Lukas Märdian (slyon)]

This sounds related to bug #2097808

[Timo Aaltonen (tjaalton)]

Hello Benjamin, or anyone else affected,

Accepted initramfs-tools into oracular-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.142ubuntu34.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-oracular to verification-done-oracular. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-oracular. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Benjamin, or anyone else affected,

Accepted initramfs-tools into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.142ubuntu25.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Fabio Augusto Miranda Martins (fabio.martins)]

I've tested the Noble and Oracular from -proposed and my feedback is positive.

Oracular with initramfs-tools 0.142ubuntu34.1:

ubuntu@fabio-oracular-amd-vanilla:~$ sudo dpkg -l | grep initramfs-tools
ii initramfs-tools 0.142ubuntu34.1 all generic modular initramfs generator (automation)
ii initramfs-tools-bin 0.142ubuntu34.1 amd64 binaries used by initramfs-tools
ii initramfs-tools-core 0.142ubuntu34.1 all generic modular initramfs generator (core tools)

Results: After a long time attempting to boot, it fails to identify the root disk (iscsi) and breaks into a initramfs shell

Full console output: https://pastebin.ubuntu.com/p/G5cHPdkjDb/

--

Noble with initramfs-tools 0.142ubuntu25.4:

ubuntu@fabio-noble-amd-vanilla:~$ sudo dpkg -l | grep initramfs-tools
ii initramfs-tools 0.142ubuntu25.4 all generic modular initramfs generator (automation)
ii initramfs-tools-bin 0.142ubuntu25.4 amd64 binaries used by initramfs-tools
ii initramfs-tools-core 0.142ubuntu25.4 all generic modular initramfs generator (core tools)

Results: After a long time attempting to boot, it fails to identify the root disk (iscsi) and breaks into a initramfs shell

Full console output: https://pastebin.ubuntu.com/p/JCBqMq8hNZ/

--

Oracular with initramfs-tools 0.142ubuntu34.2:

ubuntu@fabio-oracular-amd-proposed-native-ipv6singlestack:~$ sudo dpkg -l | grep initramfs-tools
ii initramfs-tools 0.142ubuntu34.2 all generic modular initramfs generator (automation)
ii initramfs-tools-bin 0.142ubuntu34.2 amd64 binaries used by initramfs-tools
ii initramfs-tools-core 0.142ubuntu34.2 all generic modular initramfs generator (core tools)

Results: Boots fine and quickly

Full console output: https://pastebin.ubuntu.com/p/QqC3nKXZb4/

--

Noble with initramfs-tools 0.142ubuntu25.5:

ubuntu@fabio-noble-amd-proposed-native-ipv6singlestack:~$ sudo dpkg -l | grep initramfs-tools
ii initramfs-tools 0.142ubuntu25.5 all generic modular initramfs generator (automation)
ii initramfs-tools-bin 0.142ubuntu25.5 amd64 binaries used by initramfs-tools
ii initramfs-tools-core 0.142ubuntu25.5 all generic modular initramfs generator (core tools)

Results: Boots fine and quickly

https://pastebin.ubuntu.com/p/dkRpDxXqRb/

--

PS: The "UBSAN: shift-out-of-bounds" call trace (seen even in the successful boots), is something completely unrelated and being addressed by:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2097824

So it can be safely ignored.

I'm marking verification-done.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted initramfs-tools (0.142ubuntu34.2) for oracular have finished running.
The following regressions have been reported in tests triggered by the package:

casper/1.502 (amd64)
mandos/1.8.17-1 (ppc64el)
multipath-tools/0.9.9-1ubuntu3 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/oracular/update_excuses.html#initramfs-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted initramfs-tools (0.142ubuntu25.5) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.142ubuntu25.5 (amd64, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#initramfs-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.142ubuntu25.5

---------------
initramfs-tools (0.142ubuntu25.5) noble; urgency=medium

  * configure_networking: Configure IPv4 or IPv6 based on iBFT IP address
    (LP: #2091904)
  * test:
    - d/tests/check-log: import improvements from 0.145ubuntu1
    - support setting a disk label in build_fs_ext2
    - test-common: move disc logic out of _run_qemu
    - move wait_for_server_startup to test-common
    - check for QEMU PID in wait_for_server_startup
    - Add qemu-net-iscsi autopkgtest for iBFT test cases

 -- Benjamin Drung <email address hidden> Fri, 07 Feb 2025 15:21:59 +0100

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for initramfs-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.142ubuntu34.2

---------------
initramfs-tools (0.142ubuntu34.2) oracular; urgency=medium

  * configure_networking: Configure IPv4 or IPv6 based on iBFT IP address
    (LP: #2091904)
  * test:
    - support setting a disk label in build_fs_ext2
    - test-common: move disc logic out of _run_qemu
    - move wait_for_server_startup to test-common
    - check for QEMU PID in wait_for_server_startup
    - Add qemu-net-iscsi autopkgtest for iBFT test cases

 -- Benjamin Drung <email address hidden> Fri, 07 Feb 2025 11:03:37 +0100