22.04 autoinstall crashes while installing external packages if system has two IPs on same link

[Ian Collier]

This is closely related to bug 1970409; but that bug is fixed (or claims to be) and indeed it looks to be the case that "updating packages on target system" job completes successfully if there are no extra packages to install. However, if the autoinstall config specifies extra packages to install, the installer crashes at that point.

I am currently using the 22.04.1 server install iso and have verified that the commit from bug 1970409 is present in the installer. The install is taking place on a VM that does not have direct access to the Internet and so must use a proxy. The proxy is added to the user-data YAML file for the autoinstall by a script that runs at "early-commands" stage.

The syslog from the crashed install contains this line:

subiquity_log.2258[2675]: write apt proxy info to /tmp/tmp9e00ya_m/mount/etc/apt/apt.conf.d/90curtin-aptproxy

The proxy configuration is indeed present at that location after the crash. (However, at no point during the install does the proxy info appear in /target/etc/apt/apt.conf.d.)

The installer crashes with a horrible message [insert rant about the fact that subiquity doesn't have decent error reporting] of the form:

subiquity/Install/install/postinstall: Command: '['systemd-run',.......'system-install', '/target', '--', 'ubuntu-desktop']' returned non-zero exit status 100.

We have to dig into /var/log/syslog on the crashed installer to find the relevant message (it's not even in the crash dump that the installer saves):

subiquity_log.2258[17702]: E: Unable to locate package ubuntu-desktop

What may also be relevant (although this install does work on a system that isn't proxied) is that /target/etc/apt/sources.list only contains:

deb [check-date=no] file:///cdrom jammy main restricted

If I remove the "packages:" section from the autoinstall YAML file then the installation succeeds, and indeed the proxy setting *is* present in /etc/apt/apt.conf.d on the installed system.

[Ian Collier (imc0)]

Hmm, there's not really much network activity either via the proxy
or bypassing the proxy before the installer crashes. So it seems
that the installer is believing some cache that tells it the package
does not exist.

But the exact same config, minus the proxy, works on a directly
Internet-connected VM.

[Michael Hudson-Doyle (mwhudson)]

Yes the fact that some useful output does not end up in the crash dump is a bad bug.

This sounds like the installer didn't think the network was available at all during the install, and so did not enable any network apt sources. As the pool on the ISO does not contain ubunut-desktop, this would explain the error.

The criterion for "network is available" is "there is a default route". Does your VM have a default route configured? (Either statically or from DHCP / RA). If you can attach a crash dump or the subiquity-server-debug.log file I might be able to see what's going on.

[Ian Collier (imc0)]

Thanks for your insights. The system is on DHCP and the DHCP server does always supply a default route for the connection. However, if I specify the IP/gateway/DNS on the boot commandline instead of using DHCP, the install succeeds. So this is definitely no longer related to the proxy setting, but is something weird about the network that it is on.

I've looked into this further and discovered that our DHCP server is not always returning the same IP address. In particular: during early boot the system gets an IP address (call it A1) in order to fetch the ISO image from a web server. When the system then mounts the ISO image and launches init from there, the system runs DHCP again and gets a second address (A2). The situation now is that the one ethernet interface has two IP addresses with A1 being considered primary and A2 secondary; but the default route has source A2, and the only route with source A1 is link-local.

So my speculation would be that the installer is looking for a default route with source A1 (being the primary IP of the network interface) and not finding it, therefore deciding that the system isn't networked. But networking on the system actually works fine with this setup.

The logic seems a bit funky though, because if we have a proxy we don't actually need a default route (provided the proxy and DNS servers are on the link-local subnet). We've successfully installed Ubuntu 20.04 in this same environment with an almost identical YAML file.

Anyway, the chief difference between the successful install (with one IP address) and the unsuccessful one (with two IP addresses) seems to be the presence of 'original.list' in sources.list.d - which I can see does depend on the property 'has_network', which depends on the default routes.

I tried removing the A1 address from the interface during early-commands, but it didn't make the install succeed.

[Илия Анастасов (ianastasov)]

I've encountered a similar issue when using Jammy with autoinstall on a DHCP-enabled network. In my case, I was able to resolve the issue by adding 'network-config=disabled' as a boot parameter to prevent cloud-init from interfering with the network configuration.

Update: Scratch that, even with the "network-config=disabled" boot parameter, the installer occasionally encounters the same issue. I can confirm that this happens intermittently on installations with a network configuration that includes a bridge, as shown below:

network:
  bridges:
    br0:
      addresses:
        - 192.168.1.2/24
      routes:
        - to: default
          via: 192.168.1.1
      interfaces:
        - eth0
      link-local: []
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4
        search:
          - example.com
  ethernets:
    eth0:
      dhcp4: false
      dhcp6: false
  version: 2

However, the issue does not occur on setups like this:

network:
  ethernets:
    eth0:
      addresses:
        - 192.168.1.2/24
      routes:
        - to: default
          via: 192.168.1.1
      nameservers:
        addresses:
          - 8.8.8.8
          - 8.8.4.4
        search:
          - example.com
  version: 2

[Ian Collier (imc0)]

Hmm. If I say 'network-config=disabled' then it doesn't configure any DNS servers on the cloud-init side, so it fails at even downloading the autoinstall script.