[SRU] [Jetson Orin Nano] [Jetson Agx Orin] [Jetson Orin Nx] [Jetson Agx Thor] Issues when selecting the persistent storage on Jetson devices.

[Arturo Esteban Vazquez Gutierrez]

[ Impact ]

* This is a change for hardware enablement.

* On the NVIDIA Jetson Orin platforms (Nano, NX, and AGX) installed with candidate Ubuntu for Jetson images (Jammy: https://cdimage.ubuntu.com/nvidia-tegra/ubuntu-server/jammy/daily-preinstalled/), the following warning message is displayed.

  ```
  ubuntu@ubuntu:~$ sudo dmesg --decode | grep -i "unexpected backend.*efi"
  kern :warn : [ 11.399594] pstore: ignoring unexpected backend 'efi'
  ```

  This is printed by the persistent store kernel module indicating that the system is encountering an EFI-backed pstore, but it is choosing to ignore it, meaning that in our Ubuntu Kernel the `efi_pstore` module is loaded but is not being used.

  Therefore, explicitly including the following Kernel arguments: `efi_pstore.pstore_disable=1 pstore.backend=ramoops` disables the EFI back end for the `pstore` driver and selects `ramoops` as the active back end for `pstore`.

* On the NVIDIA Jetson AGX Thor platforms installed with candidate Ubuntu for Jetson images (Noble: https://cdimage.ubuntu.com/nvidia-tegra/ubuntu-server/noble/daily-preinstalled/), the following warning message is displayed.

  ```
  ubuntu@ubuntu:~$ sudo dmesg --decode | grep -i "efi_pstore.*ramoops"
  kern :warn : [ 3.399717] pstore: backend 'efi_pstore' already in use: ignoring 'ramoops'
  ```

  Contrary to Jammy, on the Noble Thor platforms, `ramoops` is ignored, meanwhile the `efi_pstore` is used instead.

* Setting `efi_pstore.pstore_disable=1 pstore.backend=ramoops` as part of the Kernel command line arguments, on both Jetson Orin and Thor, standardizes the persistent storage for each hardware variant, avoiding the previously mentioned warnings to be displayed.

[ Test Plan ]

1. Install the candidate Ubuntu for Jetson image on a Jetson Orin device (Jammy: https://cdimage.ubuntu.com/nvidia-tegra/ubuntu-server/jammy/daily-preinstalled/manual/) and on a Jetson AGX Thor device Noble: (https://cdimage.ubuntu.com/nvidia-tegra/ubuntu-server/noble/daily-preinstalled/manual/).

2. Run the following commands to monitor the `dmesg` on the Jetson Orin device running a Jammy image, the corresponding Kernel warnings will be shown.

   ```
   ubuntu@ubuntu:~$ sudo dmesg --decode | grep -i "unexpected backend.*efi"
   kern :warn : [ 11.399594] pstore: ignoring unexpected backend 'efi'
   ```

3. Run the following commands to monitor the `dmesg` on the Jetson AGX Thor device running a Noble image, the corresponding Kernel warnings will be shown.

   ```
   ubuntu@ubuntu:~$ sudo dmesg --decode | grep -i "efi_pstore.*ramoops"
   kern :warn : [ 3.399717] pstore: backend 'efi_pstore' already in use: ignoring 'ramoops'
   ```

4. Proceed installing the updated "nvidia-tegra-defaults" package on both Orin and Thor, and reboot the devices.

5. Monitor the `dmesg` on the Jetson AGX Orin, Jetson Orin Nano/NX (Jammy), the persistent storage will be standardized, no warnings will be shown, and `ramoops` will be registered as the persistent store back end.

   ```
   ubuntu@ubuntu:~$ sudo dmesg --decode | grep -i efi
   kern :info : [ 0.000000] efi: EFI v2.70 by EDK II
   kern :info : [ 0.000000] efi: RTPROP=0x82858e698 TPMFinalLog=0x819060000 SMBIOS=0xffff0000 SMBIOS 3.0=0x8281b0000 MEMATTR=0x82192a018 ESRT=0x8219bdc18 MOKvar=0x8280e0000 TPMEventLog=0x818887018 RNG=0x818a80018 MEMRESERVE=0x819071298
   kern :notice: [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-1020-nvidia-tegra-igx root=UUID=e830c2e7-4f1b-471f-a112-825251b22500 ro fbcon=map:0 video=efifb:off console=ttyAMA0,115200 console=ttyTCU0,115200 console=tty0 quiet
   splash efi_pstore.pstore_disable=1 pstore.backend=ramoops vt.handoff=7
   kern :info : [ 0.010758] Remapping and enabling EFI services.
   kern :info : [ 0.170597] Registered efivars operations
   kern :info : [ 2.924927] integrity: Loading X.509 certificate: UEFI:MokListRT (MOKvar table)
   daemon:info : [ 5.248890] systemd[1]: Starting Load Kernel Module efi_pstore...
   daemon:info : [ 5.260792] systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
   daemon:info : [ 5.261188] systemd[1]: Finished Load Kernel Module efi_pstore.

   ubuntu@ubuntu:~$ sudo dmesg --decode | grep -i ramoops
   kern :notice: [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-1020-nvidia-tegra-igx root=UUID=e830c2e7-4f1b-471f-a112-825251b22500 ro fbcon=map:0 video=efifb:off console=ttyAMA0,115200 console=ttyTCU0,115200 console=tty0 quiet
   splash efi_pstore.pstore_disable=1 pstore.backend=ramoops vt.handoff=7
   kern :info : [ 0.136387] printk: console [ramoops-1] enabled
   kern :info : [ 0.136391] pstore: Registered ramoops as persistent store backend
   kern :info : [ 0.136396] ramoops: using 0x200000@0x82cdf0000, ecc: 0
   kern :info : [ 2.856872] printk: console [ramoops-1]: printing thread started
   ```

6. Monitor the `dmesg` on the Jetson AGX Thor (Noble), the persistent storage will be standardized, no warnings will be shown, and `ramoops` will be registered as the persistent store back end.

   ```
   ubuntu@ubuntu:~$ sudo dmesg --decode | grep -i efi
   kern :info : [ 0.000000] efi: EFI v2.7 by EDK II
   kern :info : [ 0.000000] efi: RTPROP=0x1fbb57e218 TPMFinalLog=0x1fac140000 SMBIOS=0xffff0000 SMBIOS 3.0=0x1fbb140000 MEMATTR=0x1fb6111018 ESRT=0x1fb6221798 MOKvar=0x1fbace0000 TPMEventLog=0x1f9a178018 INITRD=0x1fabfb1798 RNG=0x1fa9d98018 MEMRESERVE=0x1fabfb1218
   kern :notice: [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-1013-nvidia-tegra root=UUID=8bee1cfa-ca64-45e7-9eaf-f2e40a47df83 ro fbcon=map:0 video=efifb:off console=ttyAMA0,115200 console=ttyTCU0,115200 console=ttyUTC0,115200 console=tty0 clk_ignore_unused quiet splash vt.handoff=7 efi_pstore.pstore_disable=1 pstore.backend=ramoops
   kern :info : [ 0.002464] Remapping and enabling EFI services.
   kern :info : [ 0.043911] efivars: Registered efivars operations
   daemon:info : [ 3.725682] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
   daemon:info : [ 3.739012] systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
   daemon:info : [ 3.739212] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
   kern :info : [ 8.767494] nvidia-modeset: GPU:0: Found active displayID: 0x100 initialized by UEFI on head 0

   ubuntu@ubuntu:~$ sudo dmesg --decode | grep -i ramoops
   kern :info : [ 0.000000] OF: reserved mem: 0x0000001fbf880000..0x0000001fbfa7ffff (2048 KiB) nomap non-reusable ramoops_carveout
   kern :notice: [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-6.8.0-1013-nvidia-tegra root=UUID=8bee1cfa-ca64-45e7-9eaf-f2e40a47df83 ro fbcon=map:0 video=efifb:off console=ttyAMA0,115200 console=ttyTCU0,115200 console=ttyUTC0,115200 console=tty0 clk_ignore_unused quiet splash vt.handoff=7 efi_pstore.pstore_disable=1 pstore.backend=ramoops
   kern :info : [ 4.469340] printk: legacy console [ramoops-1] enabled
   kern :info : [ 4.469781] pstore: Registered ramoops as persistent store backend
   kern :info : [ 4.469784] ramoops: using 0x200000@0x1fbf880000, ecc: 0
   ```

7. Confirm that the persisten storage backend is enabled and set by runnig the following command.

   ```
   ubuntu@ubuntu:~$ mount | grep pstore
   pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
   ```

[ Where problems could occur ]

* Errors might occur if a user install the updated tegra-defaults package in a platform which does not have Kernel support for `PSTORE_RAM`.

  ```
  ubuntu@ubuntu:~$ grep -i CONFIG_.*PSTORE_RAM.* /boot/config-$(uname -r)
  CONFIG_PSTORE_RAM=y
  ```

* No additional problems should happen since this fix only sets the same persistent storage for the Jetson AGX Thor and the Jetson AGX Orin, Jetson Orin Nano/NX devices.

[ Other Info ]

* The inclusion of these parameters in the Kernel command line was discussed in the following private bug: https://bugs.launchpad.net/riverside/+bug/2121459

[Arturo Esteban Vazquez Gutierrez (esteban-vazquez)]

Here is the /proc/cpuinfo of the Device Under Test.

[Arturo Esteban Vazquez Gutierrez (esteban-vazquez)]

On this post there is attached a deb-diff for the corresponding fix.

The PPA test builds can be consulted through the following link:

https://launchpad.net/~esteban-vazquez/+archive/ubuntu/lp2142365

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "nvidia-tegra-defaults_1.7_1.8.diff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Frank Heimes (fheimes)]

Hi Esteban,
I had a look at the diff (and also the resulting v1.8 package) and I think it looks good.

For this version for resolute you could think about updating the Standards-Version field in d/control from "4.6.0.1" to the latest version "4.7.2" (while checking the docs about the changes: https://www.debian.org/doc/debian-policy/upgrading-checklist.html).
(But change the Standards-Version only in the development release / resolute, but leave it as is for the SRUs.)

Other than that I only see two eXperimental messages on the binaries / DEBs, which is ok.

[Arturo Esteban Vazquez Gutierrez (esteban-vazquez)]

Hello Frank,

Thank you very much for your feedback!

As suggested, I updated the Standards-Version field in d/control from "4.6.0.1" to "4.7.2", only for our Resolute package.

Please find attached on this post the corresponding debdiffs for Jammy, Noble, Questing and Resolute.

The corresponding PPA test builds can be consulted through the following link:

https://launchpad.net/~esteban-vazquez/+archive/ubuntu/lp-2142365

[Arturo Esteban Vazquez Gutierrez (esteban-vazquez)]

Hello Team,

Sorry for the inconvenience, we've been running additional validations in our DUT's and saw that the ramoops pstore is not responding as expected on the Jetson AGX Thor devices running a Noble image, currently we are reviewing this, but in the meantime lets please keep this SRU on hold (specifically the Noble, Questing and Resoulute packages) until we resolve the reported issues.