ubuntu-24.04 version: cmd-curthooks/:Fail:Installing packages on target system:efibootmgr, grub-efi-amd64-signed,nvme-cli,nvme-stas,shim-signed

[wang]

verison：ubuntu 24.04 subiquity version
OS:noble-live-server-amd64-0222.iso
CPU:AMD EPYC 9634 84-Core Processor*1
MEMORY: M321R2GA3BB6-CQKVG*2
ThinkSystem E1.S 5.9mm 7450 PRO 3.84TB Read Intensive NVMe PCIe 4.0 x4 HS SSD
UEFI MODE

lsb_release -rd: Description:Untun Noble Numbat(development branch)
Release: 24.04

What you expected to happen:
Install os on nvme,expected to install succefully.

What happened instead:
Install os on nvme, install always failed.
Before installation, we cleaned up all our nvme disk by dd if=/dev/zero of=/dev/sda bs=1M count=50000

The error log:
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: finish: cmd-install/stage-curthooks/builtin/cmd-curthooks: FAIL: Installing packages on target system: ['efibootmgr', 'grub-efi-amd64', 'grub-efi-amd64-signed', 'nvme-cli', 'nvme-stas', 'shim-signed']
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/installing-missing-packages: FAIL: installing missing packages
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: finish: cmd-install/stage-curthooks/builtin/cmd-curthooks: FAIL: curtin command curthooks
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Traceback (most recent call last):
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: File "/snap/subiquity/5511/lib/python3.10/site-packages/curtin/commands/main.py", line 202, in main
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: ret = args.func(args)
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: File "/snap/subiquity/5511/lib/python3.10/site-packages/curtin/commands/curthooks.py", line 1998, in curthooks
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: builtin_curthooks(cfg, target, state)
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: File "/snap/subiquity/5511/lib/python3.10/site-packages/curtin/commands/curthooks.py", line 1803, in builtin_curthooks
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: install_missing_packages(cfg, target, osfamily=osfamily)
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: File "/snap/subiquity/5511/lib/python3.10/site-packages/curtin/commands/curthooks.py", line 1362, in install_missing_packages
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: distro.install_packages(to_add, target=target, osfamily=osfamily)
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: File "/snap/subiquity/5511/lib/python3.10/site-packages/curtin/distro.py", line 472, in install_packages
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: return install_cmd('install', args=pkglist, opts=opts, target=target,
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: File "/snap/subiquity/5511/lib/python3.10/site-packages/curtin/distro.py", line 254, in run_apt_command
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: cmd_rv = apt_install(mode, args, opts=opts, env=env, target=target,
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: File "/snap/subiquity/5511/lib/python3.10/site-packages/curtin/distro.py", line 302, in apt_install
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: cmd_rv = inchroot.subp(cmd + dl_opts + packages, env=env,
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: File "/snap/subiquity/5511/lib/python3.10/site-packages/curtin/util.py", line 791, in subp
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: return subp(*args, **kwargs)
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: File "/snap/subiquity/5511/lib/python3.10/site-packages/curtin/util.py", line 283, in subp
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: return _subp(*args, **kwargs)
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: File "/snap/subiquity/5511/lib/python3.10/site-packages/curtin/util.py", line 147, in _subp
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: raise ProcessExecutionError(stdout=out, stderr=err,
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: curtin.util.ProcessExecutionError: Unexpected error while running command.
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Command: ['unshare', '--fork', '--pid', '--', 'chroot', '/target', 'apt-get', '--quiet', '--assume-yes', '--option=Dpkg::options::=--force-unsafe-io', '--option=Dpkg::Options::=--force-confold', 'install', '--download-only', 'efibootmgr', 'grub-efi-amd64', 'grub-efi-amd64-signed', 'nvme-cli', 'nvme-stas', 'shim-signed']
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Exit code: 100
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Reason: -
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Stdout: ''
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Stderr: ''
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Unexpected error while running command.
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Command: ['unshare', '--fork', '--pid', '--', 'chroot', '/target', 'apt-get', '--quiet', '--assume-yes', '--option=Dpkg::options::=--force-unsafe-io', '--option=Dpkg::Options::=--force-confold', 'install', '--download-only', 'efibootmgr', 'grub-efi-amd64', 'grub-efi-amd64-signed', 'nvme-cli', 'nvme-stas', 'shim-signed']
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Exit code: 100
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Reason: -
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Stdout: ''
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Stderr: ''
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]:
Mar 15 03:03:05 ubuntu-server subiquity_log.4279[6373]: Stderr: ''
Mar 15 03:03:05 ubuntu-server systemd[1]: run-u49.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED

[wang (dong28)]

i add the journal log to attachment, and we also test another ubuntu version, ubuntu 22.04 can install successfully.

[Olivier Gayot (ogayot)]

Hello and thank you for your bug report,

Could you share the content of the /var/log/installer directory?

Looking at the journal, it looks like apt could not connect to the archive. Is the network connected to the Internet? What was the result of mirror testing?

There is a known issue for "offline" installations using NVMe drives that should be fixed in today's daily:
https://bugs.launchpad.net/subiquity/+bug/2056730

I am tempted to mark this bug as a duplicate, although in your scenario, it looks like it's not completely offline.

```
Mar 15 03:03:03 ubuntu-server subiquity_log.4279[6373]: Could not connect to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (2620:2d:4002:1::101). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::16). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (91.189.91.82). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (91.189.91.81). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (91.189.91.83). - connect (113: No route to host) Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (113: No route to host)
Mar 15 03:03:03 ubuntu-server subiquity_log.4279[6373]: Err:14 http://archive.ubuntu.com/ubuntu noble/main amd64 python3-dasbus all 1.7-2
Mar 15 03:03:03 ubuntu-server subiquity_log.4279[6373]: Unable to connect to archive.ubuntu.com:http:
Mar 15 03:03:03 ubuntu-server subiquity_log.4279[6373]: Err:15 http://archive.ubuntu.com/ubuntu noble/main amd64 python3-libnvme amd64 1.7.1-1
Mar 15 03:03:03 ubuntu-server subiquity_log.4279[6373]: Unable to connect to archive.ubuntu.com:http:
Mar 15 03:03:03 ubuntu-server subiquity_log.4279[6373]: Err:16 http://archive.ubuntu.com/ubuntu noble/main amd64 python3-pyudev all 0.24.0-1
Mar 15 03:03:03 ubuntu-server subiquity_log.4279[6373]: Unable to connect to archive.ubuntu.com:http:
Mar 15 03:03:03 ubuntu-server subiquity_log.4279[6373]: Err:17 http://archive.ubuntu.com/ubuntu noble/main amd64 nvme-stas all 2.3.1-1
Mar 15 03:03:03 ubuntu-server subiquity_log.4279[6373]: Unable to connect to archive.ubuntu.com:http:
Mar 15 03:03:03 ubuntu-server subiquity_log.4279[6373]: E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/n/nvme-cli/nvme-cli_2.7.1-1_amd64.deb Could not connect to archive.ubuntu.com:80 (2620:2d:4002:1::102). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (2620:2d:4000:1::19). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (2620:2d:4002:1::103). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (2620:2d:4002:1::101). - connect (101: Network is unreachable) Could not connect to archive.ubuntu.com:80 (26...

[wang (dong28)]

Hi Olivier Gayot,
Thanks for supporting this case.

>>Could you share the content of the /var/log/installer directory?
yes, i will provide this directory later.

>>Looking at the journal, it looks like apt could not connect to the archive. Is the network connected to the Internet? What was the result of mirror testing?
Network didn't connect to Internet, we use the default configuration install ubuntu os. When installing os, it indeed wants to connect to the network, but it is not possible to download successfully.
We use the same method on other types of disk install same ubuntu version, it can be installed successfully.

Thanks

[wang (dong28)]

Upload installer log

[Dan Bungert (dbungert)]

I guess a proper nvme-o-tcp should function on a deadnet, so having the nvme-o-tcp packages in the server-ship-live seed is probably correct.

https://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/ubuntu -> server-ship-live

Desktop ISOs might consider that unnecessary.

As Olivier mentions, we're pulling in NVME-o-tcp packages for non-NVME-o-tcp cases, so that fix is on the way.

[wang (dong28)]

Hi,
Could you tell me when the fix will ready?
And how we can get the fixed version?
Our QA's test item is blocking on this issue, Will the fixed version be released in the next few days:)?

[wang (dong28)]

Tested the latest daily images at https://cdimage.ubuntu.com/ubuntu-server/daily-live/pending/.
All types of disks are good working and pass the install test now.
Request this fix patch add on next 24.04 ubuntu release.
Thanks for support!