Subiquity autoinstall fails on bare metal

[Bob Fontaine]

Install Ubuntu 22.04 on DL380 Gen 7 system with autoinstall and user-data file that works installing VMs in VSPhere and KVM/OpenStack. Install fails and offers to drop into shell. Examining /var/log/installer/subiquity-server-debug.log file shows error trying to update snap. (See attachment 1)

Subiquity user-data:

#cloud-config
autoinstall:
  identity: {hostname: titanium-server, password: removed, realname: titanium admin, username: nnadmin}
  keyboard: {layout: us, toggle: null, variant: ''}
  locale: en_US.UTF-8
  network:
    ethernets: {}
    version: 2
  ssh:
    allow-pw: true
    authorized-keys: []
    install-server: true
  storage:
    config:
    - type: disk
      id: root-disk
      ptable: gpt
      grub_device: true
      preserve: false
      wipe: superblock
      match:
        path: /dev/*da
    - {device: root-disk, size: 1G, flag: bios_grub, number: 1, preserve: false, grub_device: false, type: partition, id: partition-0}
# root partition
    - {device: root-disk, size: 5G, wipe: superblock, flag: '', number: 2, preserve: false, grub_device: false, type: partition, id: partition-1}
    - {fstype: xfs, volume: partition-1, preserve: false, type: format, id: format-1}
    - {path: /, device: format-1, type: mount, id: mount-0}
# /var
    - {device: root-disk, size: 2G, wipe: superblock, flag: '', number: 3, preserve: false, grub_device: false, type: partition, id: partition-2}
    - {fstype: xfs, volume: partition-2, preserve: false, type: format, id: format-2}
    - {path: /var, device: format-2, type: mount, id: mount-1}
# /tmp
    - {device: root-disk, size: 3G, wipe: superblock, flag: '', number: 4, preserve: false, grub_device: false, type: partition, id: partition-3}
    - {fstype: xfs, volume: partition-3, preserve: false, type: format, id: format-3}
    - {path: /tmp, device: format-3, type: mount, id: mount-2}
# /opt
    - {device: root-disk, size: 6G, wipe: superblock, flag: '', number: 5, preserve: false, grub_device: false, type: partition, id: partition-4}
    - {fstype: xfs, volume: partition-4, preserve: false, type: format, id: format-4}
    - {path: /opt, device: format-4, type: mount, id: mount-3}
# /var/log
    - {device: root-disk, size: 4G, wipe: superblock, flag: '', number: 7, preserve: false, grub_device: false, type: partition, id: partition-5}
    - {fstype: xfs, volume: partition-5, preserve: false, type: format, id: format-5}
    - {path: /var/log, device: format-5, type: mount, id: mount-4}
# /var/log/audit
    - {device: root-disk, size: 2G, wipe: superblock, flag: '', number: 8, preserve: false, grub_device: false, type: partition, id: partition-6}
    - {fstype: xfs, volume: partition-6, preserve: false, type: format, id: format-6}
    - {path: /var/log/audit, device: format-6, type: mount, id: mount-5}
# /var/tmp
    - {device: root-disk, size: 2G, wipe: superblock, flag: '', number: 9, preserve: false, grub_device: false, type: partition, id: partition-7}
    - {fstype: xfs, volume: partition-7, preserve: false, type: format, id: format-7}
    - {path: /var/tmp, device: format-7, type: mount, id: mount-6}
# /home
    - {device: root-disk, size: -1, wipe: superblock, flag: '', number: 10, preserve: false, grub_device: false, type: partition, id: partition-8}
    - {fstype: xfs, volume: partition-8, preserve: false, type: format, id: format-8}
    - {path: /home, device: format-8, type: mount, id: mount-7}

  late-commands:
    - touch /etc/cloud/cloud-init.disabled
    - /usr/bin/mkdir /target/extras
    - /usr/bin/cp -r /cdrom/netnumber/* /target/extras
    - /usr/bin/cp /target/extras/nni_log_rotate.rules /target/etc/logrotate.d/zzz-nti-logrotate
    - curtin in-target --target=/target -- apt-get -y purge cpp cpp-9 alsa* x11* strace usbmuxd
    - curtin in-target --target=/target -- apt-get -y purge landscape-common ufw git plymouth
    - curtin in-target --target=/target -- apt-get -y purge telnet lsof unattended-upgrades rsync
    - curtin in-target --target=/target -- apt-get -y purge dosfstools eatmydata fuse
    - curtin in-target --target=/target -- apt-get -y purge fonts-ubuntu-console krb5-locales mawk
    - curtin in-target --target=/target -- apt-get -y purge gsettings-desktop-schemas os-prober
    - curtin in-target --target=/target -- apt-get -y purge motd-news-config packagekit
    - curtin in-target --target=/target -- apt-get -y autoremove
    - curtin in-target --target=/target -- chmod +x /extras/CIS-compliance.sh /extras/nn-install-pkgs.sh /extras/NTI-conf.sh
    - curtin in-target --target=/target -- /extras/nn-install-pkgs.sh > /target/var/log/nn-install-pkgs.log
    - curtin in-target --target=/target -- /extras/CIS-compliance.sh > /target/var/log/CIS-compliance.log 2>&1
    - curtin in-target --target=/target -- /extras/NTI-conf.sh > /target/var/log/NTI-conf.log 2>&1
    - curtin in-target --target=/target -- echo "Please wait...will reboot automatically"
    - reboot

  version: 1

Note, when installing without autoinstall, installation succeeds.

[Dan Bungert (dbungert)]

That particular traceback should be harmless. (I'd rather we catch it then show a traceback)

If the install failed then there should be a more severe message down lower in the log.
Do you mind posting /var/log/installer/subiquity-server-debug.log / /var/log/curtin/install.log?

[Bob Fontaine (rnfontaine)]

Thanks for the quick response.

Those are the last lines in the /var/log/subiquity-server-debug.log. /var/log/curtin/install.log doesn't exist.

I forgot to mention that this install is in an air gapped environment with no access to the Ubuntu repos or an Ubuntu mirror.

[Michael Hudson-Doyle (mwhudson)]

Are there any files in /var/crash?

[Bob Fontaine (rnfontaine)]

No files in /var/crash.

One data point is that the disk was not formatted. I verified this by running the command "lsblk" and then "fdisk -l /dev/sda" which shows no partitions were created.

[Bob Fontaine (rnfontaine)]

Another data point. The console displays a message " Waiting for cloud-init...."
When the failure occurs, the display is:

cloud-init failed to complete after 10 minutes of waiting. This
suggests a bug, Which we would appreciate understanding.

The option to drop into a shell, or close is presented.

If the close option is selected, the cli version of the install is begun and will complete successfully.

The disk is 600GB. It might be possible that cloud-init is timing out due to the formatting of the disk.

[Michael Hudson-Doyle (mwhudson)]

Oh, are you using a virtual ISO/image thing from the BMC? try adding fsck.mode=skip to the kernel command line.

[Bob Fontaine (rnfontaine)]

Adding fsck.mode=skip to the kernel command line in the grub.cfg file worked.
Perhaps it should be a default option.