not able to install noble on PowerVM LPARs `Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block`

[Patricia Domingues]

After I select `Try or Install Ubuntu Server` on a PowerVM ppc64le Power9 or Power10 :

error: out of memory.

Press any key to continue...
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 6.8.0-11-generic (buildd@bos01-ppc64el-003) (powerpc64le-linux-gnu-gcc-13 (Ubuntu 13.2.0-13ubuntu1) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #11-Ubuntu SMP Wed Feb 14 00:33:03 UTC 2024 (Ubuntu 6.8.0-11.11-generic 6.8.0-rc4)
Detected machine type: 0000000000000101
command line: BOOT_IMAGE=/casper/vmlinux quiet ---
Max number of cores passed to firmware: 256 (NR_CPUS = 2048)
Calling ibm,client-architecture-support... done
memory layout at init:
  memory_limit : 0000000000000000 (16 MB aligned)
  alloc_bottom : 000000000e670000
  alloc_top : 0000000020000000
  alloc_top_hi : 0000000020000000
  rmo_top : 0000000020000000
  ram_top : 0000000020000000
instantiating rtas at 0x000000001ec30000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000010780000 -> 0x000000001078179e
Device tree struct 0x0000000010790000 -> 0x00000000107a0000
Quiescing Open Firmware ...
Booting Linux via __start() @ 0x000000000a750000 ...
[ 0.019184] plpks: POWER LPAR Platform KeyStore is not supported or enabled
[ 0.114052] SED: plpks not available
[ 0.114507] /dev/root: Can't open blockdev
[ 0.114520] List of all bdev filesystems:
[ 0.114523] ext3
[ 0.114523] ext2
[ 0.114525] ext4
[ 0.114527] squashfs
[ 0.114529] vfat
[ 0.114531] fuseblk
[ 0.114532]
[ 0.114535] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 0.114540] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.8.0-11-generic #11-Ubuntu
[ 0.114545] Hardware name: IBM,9080-HEX POWER10 (raw) 0x800200 0xf000006 of:IBM,FW1030.00 (NH1030_017) hv:phyp pSeries
[ 0.114551] Call Trace:
[ 0.114553] [c000000006403b40] [c0000000016b2f28] dump_stack_lvl+0x70/0xb4 (unreliable)
[ 0.114562] [c000000006403b70] [c0000000001926ec] panic+0x300/0x524
[ 0.114568] [c000000006403c10] [c00000000301146c] mount_root_generic+0x208/0x448
[ 0.114574] [c000000006403ce0] [c0000000030118d8] prepare_namespace+0x98/0x430
[ 0.114580] [c000000006403d70] [c000000003010820] kernel_init_freeable+0x32c/0x37c
[ 0.114585] [c000000006403de0] [c0000000000115ec] kernel_init+0x34/0x298
[ 0.114591] [c000000006403e50] [c00000000000dfbc] ret_from_kernel_user_thread+0x14/0x1c
[ 0.114596] --- interrupt: 0 at 0x0
[ 0.117006] pstore: backend (nvram) writing error (-1)
[ 0.119362] Rebooting in 10 seconds..

This is the message from a Power10(9080-HEX), but the same is happening on a PowrerVM-Power9 (9009-22A)

[Patricia Domingues (patriciasd)]

This is happening with Kernel 6.8 (currently available image noble-live-server-ppc64el.iso `2024-03-05 09:03`). I was able to install noble on the same systems last week, when Noble still had kernel 6.6.

I'm also able to install Jammy 22.04.4 on both mentioned LPARs - P9 and P10.

The issue is not happening on a Power9 bare metal.

[Patricia Domingues (patriciasd)]

re-tested with today's image http://cdimage.ubuntu.com/ubuntu-server/daily-live/20240307/ - same

[Paolo Pisati (p-pisati)]

Can you update a working image to the latest bootstrap kernel and try again?

https://launchpad.net/~canonical-kernel-team/+archive/ubuntu/bootstrap - 6.8.0-15.15

[Patricia Domingues (patriciasd)]

Paolo,
I was able to install it (image from March 1st `20240301`) , update to the bootstrap kernel from the PPA you sent:

patricia@noble-1mar:~$ cat /var/log/installer/media-info
Ubuntu-Server 24.04 LTS "Noble Numbat" - Daily ppc64el (20240301)patricia@noble-1mar:~$

patricia@noble-1mar:~$ apt list --upgradable |grep linux-gen
linux-generic/noble 6.8.0-15.15 ppc64el [upgradable from: 6.8.0-11.11+1]

...
and boot:
patricia@noble-1mar:~$ uname -a
Linux noble-1mar 6.8.0-15-generic #15-Ubuntu SMP Mon Mar 4 12:38:17 UTC 2024 ppc64le ppc64le ppc64le GNU/Linux

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/2056491

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-03-19 14:32 EDT-------
error: out of memory.

Press any key to continue...

The above messages are likely coming from the bootloader (grub2). Which version of grub2 is being used? How exactly is the install being done, e.g. via USB or some other method?

Can you try booting into the openfirmware prompt (Press 8 after restarting or activating the LPAR) and print out the real-base value? It should hopefully be set to the default value of 0xC00000.

0 > printenv real-base
-------------- Partition: common -------- Signature: 0x70 ---------------
real-base c00000 c00000
ok

If not 0xC00000 then try setting it with

0 > setenv real-base c00000

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-03-19 14:42 EDT-------
Regarding the main issue with the unable to mount to root, do you get the same errors prior to it as in the working installation? Are the kernel commandline the same? Any other differences?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-03-19 20:42 EDT-------
*** Bug 205873 has been marked as a duplicate of this bug. ***

[Patricia Domingues (patriciasd)]

Hi Chavez, thanks for checking it. I see this output:
```
     Memory Keyboard Network Speaker ok
0 > printenv real-base
-------------- Partition: common -------- Signature: 0x70 ---------------
real-base c00000 c00000
 ok
0 >
```
I'm able to install Ubuntu Jammy 22.04 LTS into the same partitions. ISO via Virtual target device.

I don't see the same error when trying an older image - working installation.

Let me know if you need any other info.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-03-26 10:05 EDT-------
(In reply to comment #7)
> Hi Chavez, thanks for checking it. I see this output:
> ```
> Memory Keyboard Network Speaker ok
> 0 > printenv real-base
> -------------- Partition: common -------- Signature: 0x70 ---------------
> real-base c00000 c00000
> ok
> 0 >
> ```
> I'm able to install Ubuntu Jammy 22.04 LTS into the same partitions. ISO via
> Virtual target device.
>
> I don't see the same error when trying an older image - working installation.
>
> Let me know if you need any other info.

Hi Patricia,
Just wondering did you try installing Ubuntu 24.04 after resetting those real-base values? We hope 24.04 installation should also go fine.

Thanks!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-03-26 13:55 EDT-------
I tried just now with the latest ISO image available on the site on a P10 Denali LPAR and still facing the same issue.

-------------------------------------------------------------------------------
error: out of memory.

Press any key to continue...

ISO used: https://cdimage.ubuntu.com/ubuntu-server/daily-live/20240326/noble-live-server-ppc64el.iso

I am not able to print "real-base" values as well. I tried setting the value but faced the same issue even after that.

Memory Keyboard Network Speaker ok
0 > printenv real-base
ok
0 > setenv real-base c00000 ok
0 > printenv real-base
ok
0 >

[Frank Heimes (fheimes)]

Hi @kowshik.jois,
the (daily) images still have the same kernel inside (the 6.8 -11):
$ rmadison --arch=ppc64el --suite=noble,noble-proposed linux-generic
 linux-generic | 6.8.0-11.11+1 | noble | ppc64el
 linux-generic | 6.8.0-20.20+1 | noble-proposed | ppc64el
But there is now a newer kernel (-20), but it is still in -proposed.
It takes some time until it transitions to release,
and again a bit until it arrives in the ISO.
So right now we are waiting until we have the -20 in the ISO, to be able to do more testing ...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-04-02 11:43 EDT-------
Can you check if vTPM is enabled and disable it and see if that allows you to install?

[Patricia Domingues (patriciasd)]

I was able to successfully install and boot PowerVM partitions - Power9 and Power10 with the latest kernel `6.8.0-20.20` (today's image - 20240405) :

POWER10 LPAR:
```
patricia@noble-a5:~$ cat /var/log/installer/media-info
Ubuntu-Server 24.04 LTS "Noble Numbat" - Daily ppc64el (20240405)patricia@noble-a5:~$
patricia@noble-a5:~$ uname -a
Linux noble-a5 6.8.0-20-generic #20-Ubuntu SMP Mon Mar 18 11:46:05 UTC 2024 ppc64le ppc64le ppc64le GNU/Linux
patricia@noble-a5:~$ lscpu |grep Model
Model name: POWER10 (architected), altivec supported
```

and POWER9 LPAR:

```
patricia@p9g-lpar-i:~$ lscpu |grep Model && uname -a
Model name: POWER9 (architected), altivec supported
Model: 2.2 (pvr 004e 0202)
Linux p9g-lpar-i 6.8.0-20-generic #20-Ubuntu SMP Mon Mar 18 11:46:05 UTC 2024 ppc64le ppc64le ppc64le GNU/Linux
```

[Frank Heimes (fheimes)]

Fantastic, thanks for re-testing and confirming!
Closing this ticket as Fix Released.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-04-07 20:28 EDT-------
This is great news but was there something in particular y'all are aware of that fixed the issue?