[Ubu 20.04][kernel-5.4.0-12-generic][WSP-DD2.3] Unable to install Ubu 20.4 on NVMe disk

[bugproxy]

[Ubu 20.04][kernel-5.4.0-12-generic][WSP-DD2.3] Unable to install Ubu 20.4 on NVMe disk

Contact Information = <email address hidden>, <email address hidden>, <email address hidden>

---uname output---
[Ubu 20.04][kernel-5.4.0-12-generic]

Machine Type = Witherspoon DD2.3

Test Environment:
* Hardware: Witherspoon-DD2.3
* Ubuntu 20.04
* kernel: 5.4.0-12-generic #15-Ubuntu

Steps:
1. Kick start Ubu 20.4 on Witherspoon-DD2.3 baremetal
2. Select NVMe disk for installation, at the end of the install seen Grub boot loader issue

  ??????????????????????? Select and install software ???????????????????????
  ? ?

             ?? [!!] Install the GRUB boot loader on a hard disk ??
  ???????????? ? ?????????
  ? ? Unable to install GRUB in /dev/nvme0n1p1 ? ?
  ? ? Executing 'grub-install /dev/nvme0n1p1' failed. ? ?
  ? ? ? ?
  ? Running "? This is a fatal error. ? ?
  ? ? ? ?
  ???????????? <Go Back> <Continue> ? ?????????
             ? ?
             ??????????????????????????????????????????????????????

     ????????? [!!] Install the GRUB boot loader on a hard disk ??????????
     ? ?
     ? Installation step failed ?
     ? An installation step failed. You can try to run the failing item ?
     ? again from the menu, or skip it and choose something else. The ?
     ? failing step is: Install the GRUB boot loader on a hard disk ?
     ? ?
     ? <Continue> ?
     ? ?
     ?????????????????????????????????????????????????????????????????????

<Tab> moves; <Space> selects; <Enter> activates buttons

Complete install screenshot attached...

grub-install doesn't seem to install the bootloader on the nvme device.
--------------------------------------------------------------------------

Logs below -

Jan 31 16:01:51 in-target: Installing for powerpc-ieee1275 platform.^M
Jan 31 16:01:52 in-target: grub-install: error: unknown filesystem.^M
Jan 31 16:01:52 in-target: Failed: grub-install --target=powerpc-ieee1275 /dev/sda1 ^M
Jan 31 16:01:52 in-target: WARNING: Bootloader is not properly installed, system may not be bootable^M
..
..

Jan 31 16:01:57 40grub2: debug: parsing: fi
Jan 31 16:01:57 40grub2: debug: parsing: ### END /etc/grub.d/41_custom ###
Jan 31 16:01:57 50mounted-tests: debug: /usr/lib/linux-boot-probes/mounted/40grub2 succeeded
Jan 31 16:01:57 linux-boot-prober: debug: linux detected by /usr/lib/linux-boot-probes/50mounted-tests
Jan 31 16:01:57 grub-installer: info: Installing grub on '/dev/nvme0n1p1'
Jan 31 16:01:57 grub-installer: info: grub-install does not support --no-floppy
Jan 31 16:01:57 grub-installer: info: Running chroot /target grub-install --force "/dev/nvme0n1p1"
Jan 31 16:01:57 grub-installer: Installing for powerpc-ieee1275 platform.
Jan 31 16:01:58 grub-installer: grub-install: error: unknown filesystem.
Jan 31 16:01:58 grub-installer: error: Running 'grub-install --force "/dev/nvme0n1p1"' failed.

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-02-19 08:59 EDT-------
Any updates from Canonical on this defect?

[Dimitri John Ledkov (xnox)]

The system has three disks /dev/sda /dev/sdb /dev/nvme0n1
I'm not sure why, but it appears that /dev/sda has been used for installing the bootloader, at least a little. I'm not sure why, and whether it is a UI bug or grub-installer logic.

$ zgrep -e sda -e nvme syslog.gz | grep grub-install
Jan 31 16:01:48 grub-installer: info: Identified partition label for /dev/nvme0n1p2: gpt
Jan 31 16:01:48 grub-installer: info: Wiping PReP partition /dev/nvme0n1p1
Jan 31 16:01:52 in-target: Failed: grub-install --target=powerpc-ieee1275 /dev/sda1 ^M
Jan 31 16:01:57 grub-installer: info: Installing grub on '/dev/nvme0n1p1'
Jan 31 16:01:57 grub-installer: info: Running chroot /target grub-install --force "/dev/nvme0n1p1"
Jan 31 16:01:58 grub-installer: error: Running 'grub-install --force "/dev/nvme0n1p1"' failed.

@Frank & Andrew
Has this been reproduced on our power machines with nvme drives?

[Andrew Cloke (andrew-cloke)]

We have access to one system (named "bobone") with an nvme drive, which has now been allocated to Dimitri for further investigation.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-03-09 10:25 EDT-------
Kalpana - can you please try this with the Ubiquity installer:

http://cdimage.ubuntu.com/ubuntu-server/daily-live/current/

The netboot image is using the DI installer - the live cd is different.

[Dimitri John Ledkov (xnox)]

grub-installer has multiple known issues when the target boot-device/rootfs-device class is not the the first one it finds/sorts/uses by default. This is experienced most commonly with scsi/nvme, but also visible with scsi/dasd drives; scsi/mmc drives, etc.

I have not confirmed that the below workarounds work on ppc64el, but in principle they should, as they do work on other platforms:

Workaround 1) install the system with just nvme drive present, and thus remove/unplug ansy sda/sdb drives

Workaround 2) ensure that sda/sdb drives are formatted clean, and do not have any PReP boot partitions (the logs indicate that PReP partitions on sda/sdb confuse grub-installer)

Workaround 3) use the the subiquity installer, as it does not use grub-installer and is not affected by this multiple-classes of drives bug.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-03-10 10:50 EDT-------
(In reply to comment #16)
> grub-installer has multiple known issues when the target
> boot-device/rootfs-device class is not the the first one it finds/sorts/uses
> by default. This is experienced most commonly with scsi/nvme, but also
> visible with scsi/dasd drives; scsi/mmc drives, etc.
>
> I have not confirmed that the below workarounds work on ppc64el, but in
> principle they should, as they do work on other platforms:
>
> Workaround 1) install the system with just nvme drive present, and thus
> remove/unplug ansy sda/sdb drives
>
> Workaround 2) ensure that sda/sdb drives are formatted clean, and do not
> have any PReP boot partitions (the logs indicate that PReP partitions on
> sda/sdb confuse grub-installer)
>
> Workaround 3) use the the subiquity installer, as it does not use
> grub-installer and is not affected by this multiple-classes of drives bug.

If I have to try "Woraround 3)" then I can try with below ISO ?
http://cdimage.ubuntu.com/ubuntu-server/daily-live/current/focal-live-server-ppc64el.iso

[Frank Heimes (fheimes)]

Yes, the "live-server" image comes with the subiquity installer, and it's
supposed to be the installer and image for 20.04.

On Tue, Mar 10, 2020 at 4:11 PM bugproxy <email address hidden> wrote:

> ------- Comment From <email address hidden> 2020-03-10 10:50 EDT-------
> (In reply to comment #16)
> > grub-installer has multiple known issues when the target
> > boot-device/rootfs-device class is not the the first one it
> finds/sorts/uses
> > by default. This is experienced most commonly with scsi/nvme, but also
> > visible with scsi/dasd drives; scsi/mmc drives, etc.
> >
> > I have not confirmed that the below workarounds work on ppc64el, but in
> > principle they should, as they do work on other platforms:
> >
> > Workaround 1) install the system with just nvme drive present, and thus
> > remove/unplug ansy sda/sdb drives
> >
> > Workaround 2) ensure that sda/sdb drives are formatted clean, and do not
> > have any PReP boot partitions (the logs indicate that PReP partitions on
> > sda/sdb confuse grub-installer)
> >
> > Workaround 3) use the the subiquity installer, as it does not use
> > grub-installer and is not affected by this multiple-classes of drives
> bug.
>
> If I have to try "Woraround 3)" then I can try with below ISO ?
>
> http://cdimage.ubuntu.com/ubuntu-server/daily-live/current/focal-live-server-ppc64el.iso
>
> --
> You received this bug notification because you are subscribed to The
> Ubuntu-power-systems project.
> https://bugs.launchpad.net/bugs/1861655
>
> Title:
> [Ubu 20.04][kernel-5.4.0-12-generic][WSP-DD2.3] Unable to install Ubu
> 20.4 on NVMe disk
>
> Status in The Ubuntu-power-systems project:
> Triaged
> Status in grub-installer package in Ubuntu:
> New
>
> Bug description:
> [Ubu 20.04][kernel-5.4.0-12-generic][WSP-DD2.3] Unable to install Ubu
> 20.4 on NVMe disk
>
> Contact Information = <email address hidden>, <email address hidden>,
> <email address hidden>
>
> ---uname output---
> [Ubu 20.04][kernel-5.4.0-12-generic]
>
> Machine Type = Witherspoon DD2.3
>
> Test Environment:
> * Hardware: Witherspoon-DD2.3
> * Ubuntu 20.04
> * kernel: 5.4.0-12-generic #15-Ubuntu
>
> Steps:
> 1. Kick start Ubu 20.4 on Witherspoon-DD2.3 baremetal
> 2. Select NVMe disk for installation, at the end of the install seen
> Grub boot loader issue
>
> ??????????????????????? Select and install software
> ???????????????????????
> ?
> ?
>
>
> ?? [!!] Install the GRUB boot loader on a hard disk ??
> ???????????? ?
> ?????????
> ? ? Unable to install GRUB in /dev/nvme0n1p1 ?
> ?
> ? ? Executing 'grub-install /dev/nvme0n1p1' failed. ?
> ?
> ? ? ?
> ?
> ? Running "? This is a fatal error. ?
> ?
> ? ? ?
> ?
> ???????????? <Go Back> <Continue> ?
> ?????????
> ? ?
> ??????????????????????????????????????????????????????...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-03-10 11:53 EDT-------
I have tried with "live-server" image... from below location:
http://cdimage.ubuntu.com/ubuntu-server/daily-live/current/focal-live-server-ppc64el.iso

Loaded the ISO as Virtual media on Witherspoon system. This should show an entry at petitb menu like mentioned below:

[USB: sda / 2020-03-06-07-31-43-00]
* Install Ubuntu Server

Kick started the install... but installer thrown with below errors....

>>>>>>
SIGTERM received, booting...
[ 409.757063] kexec_core: Starting new kernel
[ 5.242228] nouveau 0004:04:00.0: fifo: fault 00 [VIRT_READ] at 0000000000050000 engine 27 [HOST7] client 06 [HUB/HOST] reason 02 [PTE] on channel 1 [07ff8fa000 DRM]
[ 5.242336] nouveau 0004:04:00.0: bus: MMIO read of 00000000 FAULT at 6013d4 [ TIMEOUT ]
[ 5.526283] nouveau 0004:05:00.0: fifo: fault 00 [VIRT_READ] at 0000000000050000 engine 27 [HOST7] client 06 [HUB/HOST] reason 02 [PTE] on channel 1 [07ff8fa000 DRM]
[ 5.526362] nouveau 0004:05:00.0: bus: MMIO read of 00000000 FAULT at 6013d4 [ TIMEOUT ]
[ 5.725042] nouveau 0035:03:00.0: fifo: fault 00 [VIRT_READ] at 0000000000050000 engine 27 [HOST7] client 06 [HUB/HOST] reason 02 [PTE] on channel 1 [07ff8fa000 DRM]
[ 5.725128] nouveau 0035:03:00.0: bus: MMIO read of 00000000 FAULT at 6013d4 [ TIMEOUT ]
[ 5.923066] nouveau 0035:04:00.0: fifo: fault 00 [VIRT_READ] at 0000000000050000 engine 27 [HOST7] client 06 [HUB/HOST] reason 02 [PTE] on channel 1 [07ff8fa000 DRM]
[ 5.923149] nouveau 0035:04:00.0: bus: MMIO read of 00000000 FAULT at 6013d4 [ TIMEOUT ]
[ 6.355327] sd 5:0:0:0: [sdc] tag#0 request not aligned to the logical block size
[ 6.355351] blk_update_request: I/O error, dev sdc, sector 1 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[ 6.359079] sd 5:0:0:0: [sdc] tag#0 request not aligned to the logical block size
[ 6.359094] blk_update_request: I/O error, dev sdc, sector 1 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 6.359108] Buffer I/O error on dev sdc1, logical block 0, async page read
[ 6.361443] sd 5:0:0:0: [sdc] tag#0 request not aligned to the logical block size
[ 6.361455] blk_update_request: I/O error, dev sdc, sector 1 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 6.361467] Buffer I/O error on dev sdc1, logical block 0, async page read
[ 6.366444] sd 5:0:0:0: [sdc] tag#0 request not aligned to the logical block size
[ 6.366461] blk_update_request: I/O error, dev sdc, sector 1 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 6.366473] Buffer I/O error on dev sdc1, logical block 0, async page read
[ 6.370068] sd 5:0:0:0: [sdc] tag#0 request not aligned to the logical block size
[ 6.370082] blk_update_request: I/O error, dev sdc, sector 1 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 6.370094] Buffer I/O error on dev sdc1, logical block 0, async page read
[ 6.370132] sd 5:0:0:0: [sdc] tag#0 request not aligned to the logical block size
[ 6.370157] blk_update_request: I/O error, dev sdc, sector 1 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[ 6.370178] Buffer I/O error on dev sdc1, logical block 0, async page read
[ 6.376195] sd 5:0:...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-03-11 01:55 EDT-------
May be with virtual iso attach to the system seems to be not working, I have asked lab to add usb with iso to install on nvme disk.. will update once I have the setup ready.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-03-11 09:10 EDT-------
Lab created USB with
http://cdimage.ubuntu.com/ubuntu-server/daily-live/current/focal-live-server-ppc64el.iso

Petitb menu its listed as mentioned below...

[USB: sda / 2020-03-06-07-31-43-00]
* Install Ubuntu Server

Installer started with initial language select etc.. and it failed at probing disks...

Probing for devices to install to failed [ Help ]
???????????????????????????????????????????????????????????????????????????????? <14>Mar 11 12:33:51 ec2: ############ Unfortunately probing for devices to install to failed. Please report a bug
on Launchpad, and if possible include the contents of the /var/log/installer
directory.
[ Show Error Report ]
...
...

?
? Sorry, there was a problem examining the storage devices on this ? ?
? system. ? ?
? ? ?
? [ View full report ] ? ?
? ? ?
? If you want to help improve the installer, you can send an error ? ?
? report. ? ?
? ? ?
? [ Send to Canonical ] ? ?
? ? ?
? You may be able to fix the issue by switching to a shell and ? ?
? reconfiguring the system's block devices manually. ? ?
? ? ?
? [ Switch to a shell ] ?
?

selected Full report...

ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu18
Architecture: ppc64el
CasperVersion: 1.441
CrashDB: {'impl': 'launchpad', 'project': 'subiquity'}
CurrentDmesg:
[ 0.000000] dt-cpu-ftrs: setup for ISA 3000
[ 0.000000] dt-cpu-ftrs: not enabling: system-call-vectored (disabled or unsupported by kernel)
[ 0.000000] dt-cpu-ftrs: final cpu/mmu features = 0x0001986f8f5fb1a7 0x3c006041
[ 0.000000] radix-mmu: Page sizes from device-tree:
[ 0.000000] radix-mmu: Page size shift = 12 AP=0x0
[ 0.000000] radix-mmu: Page size shift = 16 AP=0x5
[ 0.000000] radix-mmu: Page size shift = 21 AP=0x1
[ 0.000000] radix-mmu: Page size shift = 30 AP=0x2
[ 0.000000] radix-mmu: Activating Kernel Userspace Execution Prevention
[ 0.000000] radix-mmu: Activating Kernel Userspace Access Prevention
[ 0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000040000000 with 1.00 GiB pages (exec)
[ 0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000002000000000 with 1.00 GiB pages
[ 0.000000] radix-mmu: Mapped 0x0000200000000000-0x0000202000000000 with 1.0/var/crash/1583930014.343549490.disk_probe_fail.crash

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2020-03-11 09:11 EDT-------

[bugproxy (bugproxy)]

------- Comment (attachment only) From <email address hidden> 2020-03-11 09:11 EDT-------

[Andrew Cloke (andrew-cloke)]

We expect an updated subiquity installer this week.
Next step is to test this new installer in an ISO using BMC vCDROM.

[Frank Heimes (fheimes)]

I just tried an updated version of subiquity (20.03) and I could successfully complete a default netboot installation a boston system (with normal disks).
grub install worked fine - post-install reboot also worked.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-03-23 11:01 EDT-------
(In reply to comment #25)
> I just tried an updated version of subiquity (20.03) and I could
> successfully complete a default netboot installation a boston system (with
> normal disks).
> grub install worked fine - post-install reboot also worked.

Can I use below posted iso?
http://cdimage.ubuntu.com/ubuntu-server/daily-live/current/focal-live-server-ppc64el.iso

camn you confirm this iso has updated version of subiquity (20.03)?

[Frank Heimes (fheimes)]

Yes, that is the latest released version and the right URL.

[Dimitri John Ledkov (xnox)]

Please provide test results, using subiquity installer, on a system with only nvme drive, or with supplementary non-nvme drives, ideally both tests.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-04-07 02:07 EDT-------
As per Mike Ranweiler's input used kexec method to kick start the install.. but I still see the reported issue that I could not able to install on NVMe disk.

#kexec -l ./vmlinux --initrd=./initrd.gz --append=?ip=9.3.90.144::9.3.90.1:255.255.255.0:ltc-wcwsp3:enP5p1s0f0:none:9.3.1.200 url=http://cdimage.ubuntu.com/ubuntu-server/daily-live/pending/focal-live-server-ppc64el.iso?
#kexec -e

?? [!!] Install the GRUB boot loader on a hard disk ??
???????????? ? ?????????
? ? Unable to install GRUB in /dev/nvme0n1p1 ? ?
? ? Executing 'grub-install /dev/nvme0n1p1' failed. ? ?
? ? ? ?
? Running "? This is a fatal error. ? ?
? ? ? ?
???????????? <Go Back> <Continue> ? ?????????
? ?
??????????????????????????????????????????????????????

Today evening let me see I can share the system details with Mike to try himself.

[Frank Heimes (fheimes)]

@kalshett
Would it be possible to share the logs of the system where you tried subiquity which failed to install grub?
Entire /var/log would be good - and in case there is something in /var/crash, that would be good to have, too - thx.

The network should be okay during the installation, and with F1 you should see a little menu in the top right that allows you to reach the Installer shell - from there you should be able to save/scp the files ...

[Dimitri John Ledkov (xnox)]

#kexec -l ./vmlinux --initrd=./initrd.gz --append=?ip=9.3.90.144::9.3.90.1:255.255.255.0:ltc-wcwsp3:enP5p1s0f0:none:9.3.1.200 url=http://cdimage.ubuntu.com/ubuntu-server/daily-live/pending/focal-live-server-ppc64el.iso?

looks good, but can you please provide checksum of vmlinux and initrd.gz used in that test?

It looks like you are using d-i vmlinux&initrd.gz, and not the subiquity /casper/vmlinux & /casper/initrd.gz as per Frank's instructions from inside the loopmoutned iso. Becuase the screenshot you show, looks like d-i and does not look like subiquity.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-04-07 12:23 EDT-------
(In reply to comment #31)
> #kexec -l ./vmlinux --initrd=./initrd.gz
> --append=?ip=9.3.90.144::9.3.90.1:255.255.255.0:ltc-wcwsp3:enP5p1s0f0:none:9.
> 3.1.200
> url=http://cdimage.ubuntu.com/ubuntu-server/daily-live/pending/focal-live-
> server-ppc64el.iso?
>
> looks good, but can you please provide checksum of vmlinux and initrd.gz
> used in that test?
>
> It looks like you are using d-i vmlinux&initrd.gz, and not the subiquity
> /casper/vmlinux & /casper/initrd.gz as per Frank's instructions from inside
> the loopmoutned iso. Becuase the screenshot you show, looks like d-i and
> does not look like subiquity.

Yes you are right, vmlinux and initrd.gz are not from /casper/ mounted. However Mike Ranweiler used both from /casper/ and facing comment #21 issue.

He may update the defect later, he is still debugging.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-04-07 13:06 EDT-------
We did this from petitboot (the boot loader), the current daily-pending iso was loop mounted and available:

# wget http://ltc-wspoon3.aus.stglabs.ibm.com/mnt/casper/vmlinux
Connecting to ltc-wspoon3.aus.stglabs.ibm.com (9.40.193.87:80)
vmlinux 100% |********************************| 26.8M 0:00:00 ETA
# wget http://ltc-wspoon3.aus.stglabs.ibm.com/mnt/casper/initrd.gz
Connecting to ltc-wspoon3.aus.stglabs.ibm.com (9.40.193.87:80)
initrd.gz 100% |********************************| 82.3M 0:00:00 ETA
# md5sum vmlinux initrd.gz
c3dcf1cff9d0f6cc649e8ce6cb9066eb vmlinux
185ba7de2818fc6ee47931342cd89f4d initrd.gz
# kexec -l ./vmlinux --initrd=./initrd.gz --append="ip=9.3.90.144::9.3.90.1:255.
255.255.0:ltc-wcwsp3:enP5p1s0f0:none:9.3.1.200 url=http://cdimage.ubuntu.com/ubu
ntu-server/daily-live/pending/focal-live-server-ppc64el.iso --quiet"
#

With this we aren't getting as far, when the iso comes up to the language selection screen I get:

[ ERROR: hook /usr/share/apport/general-hooks/powerpc.py crashed:
[ Asturianu ? ] Traceback (most recent call last):
[ Catal? ? ] File "/snap/subiquity/1625/usr/lib/python3/dist-packages/apport/report.py", line 218, in _run_hook
[ Nederlands ? ] symb['add_info'](report, ui) [ Suomi ? ]
File "/usr/share/apport/general-hooks/powerpc.py", line 42, in add_info
[ Deutsch ? ] is_kernel = report['ProblemType'].startswith('Kernel') or 'linux' in report.get('Package')]
[ Magyar ? TypeError: argument of type 'NoneType' is not iterable
[ Latvie?u ? ] ERROR: hook /usr/share/apport/general-hooks/powerpc.py crashed: ]
[ Polski ?Traceback (most recent call last):
[ ??????? ? ] File "/snap/subiquity/1625/usr/lib/python3/dist-packages/apport/report.py", line 218, in _run_hook
[ ?????????? ? ] symb['add_info'](report, ui)
File "/usr/share/apport/general-hooks/powerpc.py", line 42, in add_info
is_kernel = report['ProblemType'].startswith('Kernel') or 'linux' in report.get('Package')e
TypeError: argument of type 'NoneType' is not iterablee

Ubuntu Focal Fossa (development branch) ubuntu-server hvc0

If I go and view error reports:
? Select an error report to view: ?
? ?
? DATE KIND STATUS ?
? [ Tue Apr 7 16:59:08 2020 Disk probe failure UNVIEWED ] ?
? [ Tue Apr 7 16:59:08 2020 Block device probe failure UNVIEWED ] ?
? [ ??? Disk probe failure UNVIEWED ] ?...

[Frank Heimes (fheimes)]

Thx Michael for giving it a try.
So that is an issue that we recently also ran into.
Means the issue described by you in comment #27 is addressed in this Bug:
https://bugs.launchpad.net/ubuntu-power-systems/+bug/1867780
And it is currently looked at ...

Once this is done we can double check the grub install later in the sequence (hence I'm not marking that as duplicate for now ...).

[Dimitri John Ledkov (xnox)]

Yes, I have managed to reproduce this recent regression in KVM. And fixes
are currently awaiting review. Unfortunately it means, we need to pause
testing for this issue until probert & apport updates land on the daily iso
in the coming days. Hopefully before we the long 4 day weekend due to
public holidays.

On Tue, 7 Apr 2020, 18:16 bugproxy, <email address hidden> wrote:

> ------- Comment From <email address hidden> 2020-04-07 13:06 EDT-------
> We did this from petitboot (the boot loader), the current daily-pending
> iso was loop mounted and available:
>
> # wget http://ltc-wspoon3.aus.stglabs.ibm.com/mnt/casper/vmlinux
> Connecting to ltc-wspoon3.aus.stglabs.ibm.com (9.40.193.87:80)
> vmlinux 100% |********************************| 26.8M
> 0:00:00 ETA
> # wget http://ltc-wspoon3.aus.stglabs.ibm.com/mnt/casper/initrd.gz
> Connecting to ltc-wspoon3.aus.stglabs.ibm.com (9.40.193.87:80)
> initrd.gz 100% |********************************| 82.3M
> 0:00:00 ETA
> # md5sum vmlinux initrd.gz
> c3dcf1cff9d0f6cc649e8ce6cb9066eb vmlinux
> 185ba7de2818fc6ee47931342cd89f4d initrd.gz
> # kexec -l ./vmlinux --initrd=./initrd.gz --append="ip=9.3.90.144::
> 9.3.90.1:255.
> 255.255.0:ltc-wcwsp3:enP5p1s0f0:none:9.3.1.200 url=
> http://cdimage.ubuntu.com/ubu
> ntu-server/daily-live/pending/focal-live-server-ppc64el.iso
> <http://cdimage.ubuntu.com/ubuntu-server/daily-live/pending/focal-live-server-ppc64el.iso>
> --quiet"
> #
>
> With this we aren't getting as far, when the iso comes up to the
> language selection screen I get:
>
> [ ERROR: hook /usr/share/apport/general-hooks/powerpc.py crashed:
> [ Asturianu ? ] Traceback
> (most recent call last):
> [ Catal? ? ]
> File
> "/snap/subiquity/1625/usr/lib/python3/dist-packages/apport/report.py", line
> 218, in _run_hook
> [ Nederlands ? ]
> symb['add_info'](report, ui) [ Suomi
> ? ]
> File "/usr/share/apport/general-hooks/powerpc.py", line 42, in add_info
> [ Deutsch ? ]
> is_kernel = report['ProblemType'].startswith('Kernel') or 'linux' in
> report.get('Package')]
> [ Magyar ? TypeError: argument of type
> 'NoneType' is not iterable
> [ Latvie?u ? ]
> ERROR: hook
> /usr/share/apport/general-hooks/powerpc.py crashed: ]
> [ Polski ?Traceback (most recent call
> last):
> [ ??????? ? ]
> File
> "/snap/subiquity/1625/usr/lib/python3/dist-packages/apport/report.py", line
> 218, in _run_hook
> [ ?????????? ? ]
> symb['add_info'](report, ui)
> File "/usr/share/apport/general-hooks/powerpc.py", line 42, in add_info
> is_kernel = report['ProblemType'].startswith('Kernel') or 'linux' in
> report.get('Package')e
> TypeError: argument of type 'NoneType' is not iterablee
>
> Ubuntu Focal Fossa (development branch) ubuntu-server hvc0
>
> If I go and view error reports:
> ? Select an error report to...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-04-16 03:29 EDT-------
I think I've probably seen a couple different issues between this and LP 1867780 (and maybe others). What I was seeing today is that using 'Use an entire disk' in the Guided storage configuration screen with version 20.04.1 was working for me, but a 'Custom storage layout' always failed - even just reformatting existing partitions. I think that is different than what was reported last week. 20.04.1+git19.3bd1382b fixes that issue for me.

I tried an nvme install on a large multipath system, but the installer had a number of errors before failing on the partitioning (use entire disk). I did send a report on it. I'll try that again tomorrow, too. Hopefully I'll be able to try an nvme install on the original system, too.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-04-16 12:27 EDT-------
I was able to install successfully on NVMe using the updated (20.04.1+git19.3bd1382b) installer:
ubuntu@ltc-wcwsp3:~$ mount |grep nv
/dev/nvme0n1p2 on / type ext4 (rw,relatime)
ubuntu@ltc-wcwsp3:~$

Using the old (22.04.1) installer I choose 'Use entire disk' errors with:
ERROR: hook /usr/share/apport/general-hooks/ubuntu.py crashed:??????????????????
An error occurred during installation tar: Removing leadiTraceback (most recent call last):
???????????????????????????????????????????????????????????????????????????????? File "/snap/subiqui

This works on the new installer (20.04.1+git19.3bd1382b) - I can successfully use 'Use entire disk' and select the NVMe device and install.

However, I still see a problem when I use 'Custom storage layout' with NVMe on 20.04.1+git19.3bd1382b:
tar: Removing leading `/' from member names?????????????????????????????????????
An error occurred during installation tar: Removing leading `/' from hard link targets
????????????????????????????????????????????????????????????????????????????????

I did send logs, I can paste more info if that's better.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-04-17 01:59 EDT-------
I think this bug has gone through a few different actual problems to where we are now. To re-iterate, with 20.04.1+git19.3bd1382b I can successfully install on a NVMe disk when I choose 'Use entire disk' and select that disk. If I try to use 'Custom storage layout' to configure an install to the NVMe disk then the install fails. This is not specific to nvme, though, I see that with a standard disk, too. I did try that scenario in an x86 vm and it worked ok, so it looks power specific that 'Custom storage layout' doesn't work, at least from what I've seen.

[Frank Heimes (fheimes)]

Well, I assume that a new LP bug needs to be raised in this case, since the initial NVMe problem seems to be gone.
We'll check with the very latest image (timestamp April 16) - that came out very late yesterday (my day).

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-04-17 03:10 EDT-------
(In reply to comment #38)
> I think this bug has gone through a few different actual problems to where
> we are now. To re-iterate, with 20.04.1+git19.3bd1382b I can successfully
> install on a NVMe disk when I choose 'Use entire disk' and select that disk.
> If I try to use 'Custom storage layout' to configure an install to the NVMe
> disk then the install fails. This is not specific to nvme, though, I see
> that with a standard disk, too. I did try that scenario in an x86 vm and it
> worked ok, so it looks power specific that 'Custom storage layout' doesn't
> work, at least from what I've seen.

When I first reported I did selected entire disk option and reported it wasn't installed on nvme.

[Andrew Cloke (andrew-cloke)]

Thanks for your testing. Marking as "Fix Released".

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-04-20 01:11 EDT-------
(In reply to comment #39)
> Well, I assume that a new LP bug needs to be raised in this case, since the
> initial NVMe problem seems to be gone.
> We'll check with the very latest image (timestamp April 16) - that came out
> very late yesterday (my day).

I checked the 'Custom storage layout' on the 0417 pending ISO (20.04.1+git19.3bd1382b still) and it worked ok. That still is the current pending ISO, but I'll retry it as the new one gets available and open a new issue if I recreate.

[Agyemang Richmond (qwams)]

I can't install Ubuntu 20.04 due to grub fatal error

[Frank Heimes (fheimes)]

Which image are you using for the installation?
The (legacy) server image (based on d-i) or live-server (based on subiquity)?
In case you used live-server / subiquity - did you updated to the last version of the installer when asked?
And on which architecture do have that problem? (Please notice that this ticket is [actually was, because it's already closed now] about Ubuntu Server for ppc64el.)
Unfortunately your report doesn't have enough information, please also share the content of /var/log and in case not empty also the content of /var/crash.
And since this ticket is already closed - means flagged as 'Fix Released' - people will not really notice or react on updates, hence I suggest to open a new and separate ticket for the issue you faced (incl. the logs).

[Agyemang Richmond (qwams)]

Thanks for the response.

My installation medium is the Ubuntu desktop version

On Sun, 17 May 2020 at 11:10, Frank Heimes <email address hidden>
wrote:

> Which image are you using for the installation?
> The (legacy) server image (based on d-i) or live-server (based on
> subiquity)?
> In case you used live-server / subiquity - did you updated to the last
> version of the installer when asked?
> And on which architecture do have that problem? (Please notice that this
> ticket is [actually was, because it's already closed now] about Ubuntu
> Server for ppc64el.)
> Unfortunately your report doesn't have enough information, please also
> share the content of /var/log and in case not empty also the content of
> /var/crash.
> And since this ticket is already closed - means flagged as 'Fix Released'
> - people will not really notice or react on updates, hence I suggest to
> open a new and separate ticket for the issue you faced (incl. the logs).
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1861655
>
> Title:
> [Ubu 20.04][kernel-5.4.0-12-generic][WSP-DD2.3] Unable to install Ubu
> 20.4 on NVMe disk
>
> Status in The Ubuntu-power-systems project:
> Fix Released
> Status in grub-installer package in Ubuntu:
> Fix Released
> Status in subiquity package in Ubuntu:
> Fix Released
>
> Bug description:
> [Ubu 20.04][kernel-5.4.0-12-generic][WSP-DD2.3] Unable to install Ubu
> 20.4 on NVMe disk
>
> Contact Information = <email address hidden>, <email address hidden>,
> <email address hidden>
>
> ---uname output---
> [Ubu 20.04][kernel-5.4.0-12-generic]
>
> Machine Type = Witherspoon DD2.3
>
> Test Environment:
> * Hardware: Witherspoon-DD2.3
> * Ubuntu 20.04
> * kernel: 5.4.0-12-generic #15-Ubuntu
>
> Steps:
> 1. Kick start Ubu 20.4 on Witherspoon-DD2.3 baremetal
> 2. Select NVMe disk for installation, at the end of the install seen
> Grub boot loader issue
>
> ??????????????????????? Select and install software
> ???????????????????????
> ?
> ?
>
>
> ?? [!!] Install the GRUB boot loader on a hard disk ??
> ???????????? ?
> ?????????
> ? ? Unable to install GRUB in /dev/nvme0n1p1 ?
> ?
> ? ? Executing 'grub-install /dev/nvme0n1p1' failed. ?
> ?
> ? ? ?
> ?
> ? Running "? This is a fatal error. ?
> ?
> ? ? ?
> ?
> ???????????? <Go Back> <Continue> ?
> ?????????
> ? ?
> ??????????????????????????????????????????????????????
>
>
> ????????? [!!] Install the GRUB boot loader on a hard disk
> ??????????
> ?
> ?
> ? Installation step failed
> ?
> ? An installation step failed. You can try to run the failing item
> ?
> ? again from the menu, or skip it and choose something else. The
> ?
> ...