GPIO error logs in start and dmesg after update of kernel

Bug #1937897 reported by tothsoft@gmail.com
76
This bug affects 14 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Undecided
AceLan Kao
Hirsute
Medium
AceLan Kao
Impish
Undecided
AceLan Kao

Bug Description

[Impact]
After upgrade kernel to 5.11.0-25 which introduce some ODM patches from AAEON, user encounters below errors
[   5.852182] gpio gpiochip2: (gpio_aaeon): tried to insert a GPIO chip with zero lines
[ 5.852187] gpiochip_add_data_with_key: GPIOs 0..-1 (gpio_aaeon) failed to register, -22
[ 5.852194] gpio-aaeon: probe of gpio-aaeon.0 failed with error -22

[Fix]
AAEON provides a patch to check the BFPI version before loading the driver.
Which fixes the issue introduced by
Hirsute:
   45a8bb8699cc UBUNTU: ODM: mfd: Add support for IO functions of AAEON devices
Impish:
   424945128781 UBUNTU: ODM: mfd: Add support for IO functions of AAEON devices

[Test]
Verified by AAEON.

[Where problems could occur]
It adds a check while probing the driver, should have no impact to normal user.

=================================================

After update from kernel 5.11.0-22 to 5.11.0-25 i see next logs error to gpio:

   5.852182] gpio gpiochip2: (gpio_aaeon): tried to insert a GPIO chip with zero lines
[ 5.852187] gpiochip_add_data_with_key: GPIOs 0..-1 (gpio_aaeon) failed to register, -22
[ 5.852194] gpio-aaeon: probe of gpio-aaeon.0 failed with error -22

On pc:

    description: Desktop Computer
    product: System Product Name (SKU)
    vendor: ASUS
    version: System Version
    serial: System Serial Number
    width: 64 bits
    capabilities: smbios-3.3.0 dmi-3.3.0 smp vsyscall32
    configuration: boot=normal chassis=desktop family=To be filled by O.E.M. sku=SKU uuid=0ABCA172-BFA8-2AC5-FC37-3C7C3FD88FE4
  *-core
       description: Motherboard
       product: TUF GAMING B550M-PLUS (WI-FI)
       vendor: ASUSTeK COMPUTER INC.
       physical id: 0
       version: Rev X.0x
       serial: 201176738701636
       slot: Default string
     *-firmware
          description: BIOS
          vendor: American Megatrends Inc.
          physical id: 0
          version: 2403
          date: 06/16/2021
          size: 64KiB
          capacity: 16MiB
          capabilities: pci apm upgrade shadowing cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer acpi usb biosbootspecification uefi
     *-memory

CVE References

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux (Ubuntu):
status: New → Confirmed
Revision history for this message
Ole Jon Bjørkum (olejonbj) wrote (last edit ):

EDIT: To clarify both times these lines come they refer to "gpiochip1" (so not 2 different ones).

I started getting the same 3 lines of errors TWICE at boot, with the now latest official 5.8 HWE kernel. Without "splash", just "quiet" set for GRUB it also shows on my monitors when booting. Honestly seems to slow down boot too (I mean I have 5.5 GB/s R/W NVMes and it can get to GDM in maybe less than 2 seconds).

WHAT HAPPENED:

After my 20.04.2 (HWE) kernel was updated From:

linux-image-5.8.0-59-generic

To:

linux-image-5.8.0-63-generic

If I boot the former (*-59) or any older, I don't see the below errors TWICE. With the latter (*-63) I do. Seems to be little to none information about this. I only find it in the Linux source code. Is it something new in Linux?

Running journalctl -b (now) first 3 lines of errors come after early kernel NVME(s) preparation, but have just checked that once (now). Seems random as the next exact same 3 lines come below something completely different in the log. But again system can boot in 2 secs, however seems slower with these in fact.

$ journalctl -b

RED LINES:
kernel: gpio gpiochip1: (gpio_aaeon): tried to insert a GPIO chip with zero lines
gpiochip_add_data_with_key: GPIOs 0..-1 (gpio_aaeon) failed to register, -22

YELLOW LINE:
gpio-aaeon: probe of gpio-aaeon.0 failed with error -22

... and again later in the log:

RED LINES:
kernel: gpio gpiochip1: (gpio_aaeon): tried to insert a GPIO chip with zero lines
gpiochip_add_data_with_key: GPIOs 0..-1 (gpio_aaeon) failed to register, -22

YELLOW LINE:
gpio-aaeon: probe of gpio-aaeon.0 failed with error -22

---

Apart from that everything works. No new hardware has been added.

Pretty generic system (hefty CPU but still X570 + Zen2):

* ASUS X570 AM4 Motherboard ("Prime Pro")
* Ryzen Zen2 3950X 16C/32T CPU
* ASUS Radeon RX 5500 XT 8 GB OC GPU
* 2 NVMes and 3 HDDs (total 12 TB)
* No special peripherals

Happens really early at boot, so seems low-level.

Is there at least a way to hide these or disable the kernel from trying and failing? Kernel/GRUB parameter? Module parameter? Blacklist a new module? Just running "locate -i gpio", well it seems that both kernels have the same files containing the name "gpio", but haven't compared 100%. No gpio module, service or anything that wasn't there before.

Revision history for this message
Alex Hung (alexhung) wrote :

gpio-aaeon was added into newer versions, ex.

$ git log Ubuntu-5.11.0-22.23..Ubuntu-5.11.0-25.27 -- ./drivers/gpio/gpio-aaeon.c
commit 16c04b79bdc6e13a8f1c9b372b34c6f32f3806bb
Author: Kunyang_Fan <email address hidden>
Date: Wed Jun 16 07:56:00 2021 +0200

    UBUNTU: ODM: gpio: add driver for AAEON devices

    BugLink: https://bugs.launchpad.net/bugs/1929504

    This patch add support for the GPIO pins whose control are
    transported to BIOS through ASUS WMI interface.

    Signed-off-by: Kunyang_Fan <email address hidden>
    Review-by: Kai-Heng Feng <email address hidden>
    Review-by: Chia-Lin Kao (AceLan) <email address hidden>
    Signed-off-by: Chia-Lin Kao (AceLan) <email address hidden>
    Acked-by: Stefan Bader <email address hidden>
    Acked-by: Kleber Sacilotto de Souza <email address hidden>
    Signed-off-by: Stefan Bader <email address hidden>

Similarly,

$ git log Ubuntu-5.8.0-59.66..Ubuntu-5.8.0-63.71 -- ./drivers/gpio/gpio-aaeon.c
commit 6811515eb04775991d42ee7f6aec5fbd6b69c5cb
Author: Kunyang_Fan <email address hidden>
Date: Wed Jun 16 07:56:00 2021 +0200

    UBUNTU: ODM: gpio: add driver for AAEON devices

    BugLink: https://bugs.launchpad.net/bugs/1929504

    This patch add support for the GPIO pins whose control are
    transported to BIOS through ASUS WMI interface.

    Signed-off-by: Kunyang_Fan <email address hidden>
    Review-by: Kai-Heng Feng <email address hidden>
    Review-by: Chia-Lin Kao (AceLan) <email address hidden>
    Signed-off-by: Chia-Lin Kao (AceLan) <email address hidden>
    Acked-by: Stefan Bader <email address hidden>
    Acked-by: Kleber Sacilotto de Souza <email address hidden>
    Signed-off-by: Stefan Bader <email address hidden>

Revision history for this message
Alex Hung (alexhung) wrote :

I don't have a system support this module but you can try to blacklist it with

  $ echo "blacklist gpio-aaeon" | sudo tee -a /etc/modprobe.d/blacklist.conf

summary: - GPIO error logs in start and dmesg aftzer update of kernel
+ GPIO error logs in start and dmesg after update of kernel
Revision history for this message
Ole Jon Bjørkum (olejonbj) wrote :

No, I can't, because there isn't a single module loaded with even the name "gpio" in it.

So running: $ lsmod|grep gpio == No result

It's in the kernel, not a module. I reverted to the GA kernel and although with that I've had some issue with on headless *Intel* NUC running Virtual Machines that use the more optimized Ubuntu KVM kernel, using libvirt (host uses the latest HWE), where both host and VMs slow down to a crawl randomly without errors.

But on my Ryzen 9 workstation the GA kernel is super stable, fast and just works. A little slower in Vulkan *Benchmarks*, that's basically it. And yes, it runs a bunch of server services. Mainly does that like "22/7", for LAN and WAN.

Biggest "drawback" is that the 5.4 kernel's k10temp module does not display the temperatures of my Corsair PCIe4 5.5 GB/s NVMEs (just running "sensors"), but well that doesn't matter. I've stressed them well enough before, to see that my cooling is absolutely enough (and they come with a built-in heatsink.

Sure I could've reverted to the previous 5.8 kernel, but it's tied to HWE (metapackages), unless I uninstall them, and then, well no security updates.

Revision history for this message
Ole Jon Bjørkum (olejonbj) wrote :

Well OK there is a module with that name, however I did blacklist it and same error still occurs. It happens very early in the boot process. Now maybe it tries to load gpio-aaeon, but still early to load modules I would say, it's still doing say GRUB kernel parameters like fsck ones.

Now I have confirmed it does slow down boot pretty badly, however the HWE stack just got kernel 5.11 and although it has the same error it does seem to *almost* boot as fast as without any error.

Now I tried to actually load that module, so the reverse. It simply says that no such device exists.

Conclusion/TLDR: So it shouldn't try to load it in the first place.

AceLan Kao (acelankao)
description: updated
Changed in linux (Ubuntu Hirsute):
status: New → In Progress
Changed in linux (Ubuntu Impish):
status: Confirmed → In Progress
Changed in linux (Ubuntu Hirsute):
assignee: nobody → AceLan Kao (acelankao)
Changed in linux (Ubuntu Impish):
assignee: nobody → AceLan Kao (acelankao)
Revision history for this message
AceLan Kao (acelankao) wrote :

Hi,

This is the test kernel which should fix this issue, please give it a try.
Thanks.

https://people.canonical.com/~acelan/bugs/lp1937897/

Revision history for this message
pelm (pelle-ekh) wrote :

No it unfortunately did not work. With the new HWE kernel Ubuntu 5.11.0-27.29~20.04.1-generic (same as above?).

The failing lines are at two places in the log. Boot is very quick though but it is annoying seeing the red lines.

Revision history for this message
pelm (pelle-ekh) wrote :

OOps, this this isn't the same kernel!? Ignore my comment then!

Revision history for this message
Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1937897

tags: added: iso-testing
AceLan Kao (acelankao)
description: updated
Revision history for this message
Bob H (bobbicat) wrote :

These two error messages display when booting and closing live disk media using 'safe graphics'

gpio gpiochip: (gpio_aaeon): tried to insert a GPIO chip with zero lines
gpiochip_add_data_with_key: GPIOs 0..-1 (gpio_aaeon) failed to register, -22

they do not seem to upset the processes of the media

this occurs with Kubunt Impish daily builds and also with Kubuntu 2004.3 daily build

Changed in linux (Ubuntu Hirsute):
status: In Progress → Fix Committed
Changed in linux (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Revision history for this message
Stefan Bader (smb) wrote :

This is NOT fix-released, yet.

Changed in linux (Ubuntu Hirsute):
status: Fix Released → Fix Committed
importance: Undecided → Medium
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-hirsute' to 'verification-done-hirsute'. If the problem still exists, change the tag 'verification-needed-hirsute' to 'verification-failed-hirsute'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-hirsute
Revision history for this message
Ole Jon Bjørkum (olejonbj) wrote :

Added tag 'verification-failed-hirsute'. Enabled proposed and installed package 'linux-image-5.13.0-14-generic' version '5.13.0-14.14~20.04.4' on my 20.04.3 LTS, and still get in dmesg:

[ 1.228729] gpio gpiochip1: (gpio_aaeon): tried to insert a GPIO chip with zero lines
[ 1.228732] gpiochip_add_data_with_key: GPIOs 0..-1 (gpio_aaeon) failed to register, -22
[ 1.228735] gpio-aaeon: probe of gpio-aaeon.0 failed with error -22
[ 7.238390] gpio gpiochip1: (gpio_aaeon): tried to insert a GPIO chip with zero lines
[ 7.238396] gpiochip_add_data_with_key: GPIOs 0..-1 (gpio_aaeon) failed to register, -22
[ 7.238401] gpio-aaeon: probe of gpio-aaeon.0 failed with error -22

tags: added: verification-failed-hirsute
removed: verification-needed-hirsute
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
Stefan Bader (smb) wrote :

The Hirsute kernel to be verified is: 5.11.0-35.37 or the current hwe-5.11 (5.11.0-35.37~20.04.1) kernel for focal. The Impish work (5.13) is still in progress.

tags: added: verification-needed-hirsute
removed: verification-failed-hirsute
Revision history for this message
tothsoft@gmail.com (tothsoft) wrote :

On todays updated kernel 5.11.0-34-generic is next message:

[ 7.284731] gpio gpiochip2: (gpio_aaeon): tried to insert a GPIO chip with zero lines
[ 7.284753] gpiochip_add_data_with_key: GPIOs 0..-1 (gpio_aaeon) failed to register, -22

tags: added: verification-failed-focal
removed: verification-needed-focal
Revision history for this message
Ole Jon Bjørkum (olejonbj) wrote (last edit ):

To be clear this issue started with Kernel 5.8.0-63 NOT 5.11 (on Focal / 20.04 HWE LTS)

Well Ubuntu Kernel Bot, that's exactly what I did in comment #14 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1937897/comments/14

Latest 5.13 kernel in -prosed on Focal (20.04.3) gets the same gpio errors and warnings in dmesg and "journalctl -b" as with the current 5.11 HWE kernel.

X570 Motherboard
Ryzen 9 3950X Zen2 CPU

Same error appears twice during boot. Here's the output from "journalctl -b"

---
sep. 12 15:37:43 tux kernel: gpio gpiochip1: (gpio_aaeon): tried to insert a GPIO chip with zero lines
sep. 12 15:37:43 tux kernel: gpiochip_add_data_with_key: GPIOs 0..-1 (gpio_aaeon) failed to register, -22
sep. 12 15:37:43 tux kernel: gpio-aaeon: probe of gpio-aaeon.0 failed with error -22
sep. 12 15:37:44 tux kernel: gpio gpiochip1: (gpio_aaeon): tried to insert a GPIO chip with zero lines
sep. 12 15:37:44 tux kernel: gpiochip_add_data_with_key: GPIOs 0..-1 (gpio_aaeon) failed to register, -22
sep. 12 15:37:44 tux kernel: gpio-aaeon: probe of gpio-aaeon.0 failed with error -22
---

They're both "gpiochip1 - GPIOs 0..-1", so why try again right after the first fails?

Does NOT appear on my Intel NUC Core i3. It's a 2013 model but runs the exact same kernel at least. Likely not on any Intel based motherboards.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 5.13.0-16.16

---------------
linux (5.13.0-16.16) impish; urgency=medium

  * impish/linux: 5.13.0-16.16 -proposed tracker (LP: #1942611)

  * Miscellaneous Ubuntu changes
    - [Config] update toolchain in configs

  * Miscellaneous upstream changes
    - Revert "UBUNTU: [Config] Enable CONFIG_UBSAN_BOUNDS"

 -- Andrea Righi <email address hidden> Fri, 03 Sep 2021 16:21:14 +0200

Changed in linux (Ubuntu Impish):
status: In Progress → Fix Released
Revision history for this message
Ole Jon Bjørkum (olejonbj) wrote (last edit ):

And will that kernel come to the Ubuntu 20.04 LTS HWE stack, say in 20.04.4? Kinda need to use LTS versions, us that need that stability at least.

Not really interested in fixes for 21.10 or something... Focal.

Revision history for this message
Saša Marjanović (cy6ernaut) wrote (last edit ):

I've just updated to kernel 5.11.0-36.40~20.04.1 (I'm on HWE stack on Linux Mint 20.2). I'm still getting the same log errors as OP reported.

Revision history for this message
pelm (pelle-ekh) wrote :

Yes I can confirm that the updated HWE kernel 5.11.0-36.40~20.04.1-generic didn't work; same log errors.

Revision history for this message
Ole Jon Bjørkum (olejonbj) wrote :

Confirmed. Just updated and same.

Can you backport the fix to the 20.04 HWE kernel? I mean that's the one Canonical recommends has as first choice if downloading Ubuntu from ubuntu.com, for years now. And they've become more stable in my experience than the ones between LTS versions, which is good. The others should be more testing grounds and those who can live with having to fix possibly basic stuff.

And definitely more common nowadays that third party software outside repos (incl. PPAs), proprietary or not, often lists only the latest or two latest LTS versions as supported and downloads (repo or deb package).

Tried Ubuntu 21.04 some months ago, just to see if something new cool stuff, in a VM (IIRC, or it was a USB stick) and well let's say KVM+QEMU or my PC, me either, didn't like it. Boot warnings, even red errors from systemd on a clean updated install, and several (official) packages that are in the repos didn't even launch, some DKMS modules failed... Can't use that as a primary workstation + server.

With snaps etc you can run pretty bleeding edge both desktop and CLI stuff like certbot (Let's Encrypt) on 20.04, so. Snaps use an 18.04 core - makes sense for compatibility. Doesn't stop you from the latest Blender version or whatever.

Revision history for this message
Kleber Sacilotto de Souza (kleber-souza) wrote :

Hi,

Focal HWE kernel 5.11.0-36.40~20.04.1 does *not* contain the fix for this bug. The kernel version currently in focal-proposed (5.11.0-35.37~20.04.1) contains the fix but it is getting re-spun and will be updated to a new version in a couple of hours. Please test version 5.11.0-35.37~20.04.1 or any new version that will be available in -proposed at the time (likely beginning with 5.11.0-37.41).

Thank you.

tags: added: verification-needed-focal
removed: verification-failed-focal
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (68.6 KiB)

This bug was fixed in the package linux - 5.11.0-37.41

---------------
linux (5.11.0-37.41) hirsute; urgency=medium

  * hirsute/linux: 5.11.0-37.41 -proposed tracker (LP: #1944180)

  * CVE-2021-41073
    - io_uring: ensure symmetry in handling iter types in loop_rw_iter()

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2021.09.06)

  * LRMv5: switch primary version handling to kernel-versions data set
    (LP: #1928921)
    - [Packaging] switch to kernel-versions

  * disable “CONFIG_HISI_DMA” config for ubuntu version (LP: #1936771)
    - Disable CONFIG_HISI_DMA
    - [Config] Record hisi_dma no longer built for arm64

  * ubunut_kernel_selftests: memory-hotplug: avoid spamming logs with
    dump_page() (LP: #1941829)
    - selftests: memory-hotplug: avoid spamming logs with dump_page(), ratio limit
      hot-remove error test

  * alsa: the soundwire audio doesn't work on the Dell TGL-H machines
    (LP: #1941669)
    - ASoC: SOF: allow soundwire use desc->default_fw_filename
    - ASoC: Intel: tgl: remove sof_fw_filename set for tgl_3_in_1_default

  * e1000e blocks the boot process when it tried to write checksum to its NVM
    (LP: #1936998)
    - e1000e: Do not take care about recovery NVM checksum

  * Dell XPS 17 (9710) PCI/internal sound card not detected (LP: #1935850)
    - ASoC: Intel: sof_sdw: include rt711.h for RT711 JD mode
    - ASoC: Intel: sof_sdw: add quirk for Dell XPS 9710

  * mute/micmute LEDs no function on HP ProBook 650 G8 (LP: #1939473)
    - ALSA: hda/realtek: fix mute/micmute LEDs for HP ProBook 650 G8 Notebook PC

  * Fix mic noise on HP ProBook 445 G8 (LP: #1940610)
    - ALSA: hda/realtek: Limit mic boost on HP ProBook 445 G8

  * GPIO error logs in start and dmesg after update of kernel (LP: #1937897)
    - ODM: mfd: Check AAEON BFPI version before adding device

  * External displays not working on Thinkpad T490 with ThinkPad Thunderbolt 3
    Dock (LP: #1938999)
    - drm/i915/ilk-glk: Fix link training on links with LTTPRs

  * Fix kernel panic caused by legacy devices on AMD platforms (LP: #1936682)
    - SAUCE: iommu/amd: Keep swiotlb enabled to ensure devices with 32bit DMA
      still work

  * Hirsute update: upstream stable patchset 2021-08-30 (LP: #1942123)
    - drm/i915: Revert "drm/i915/gem: Asynchronous cmdparser"
    - Revert "drm/i915: Propagate errors on awaiting already signaled fences"
    - regulator: rtmv20: Fix wrong mask for strobe-polarity-high
    - regulator: rt5033: Fix n_voltages settings for BUCK and LDO
    - spi: stm32h7: fix full duplex irq handler handling
    - ASoC: tlv320aic31xx: fix reversed bclk/wclk master bits
    - r8152: Fix potential PM refcount imbalance
    - qed: fix possible unpaired spin_{un}lock_bh in _qed_mcp_cmd_and_union()
    - ASoC: rt5682: Fix the issue of garbled recording after powerd_dbus_suspend
    - net: Fix zero-copy head len calculation.
    - ASoC: ti: j721e-evm: Fix unbalanced domain activity tracking during startup
    - ASoC: ti: j721e-evm: Check for not initialized parent_clk_id
    - efi/mokvar: Reserve the table only if it is in boot services data
    - nvme: fix nvme_setup_command ...

Changed in linux (Ubuntu Hirsute):
status: Fix Committed → Fix Released
Revision history for this message
Ole Jon Bjørkum (olejonbj) wrote (last edit ):

Confirmed! Finally. With the latest HWE:

$ apt show linux-image-5.11.0-37-generic

Package: linux-image-5.11.0-37-generic
Version: 5.11.0-37.41~20.04.2
...

NO errors or warnings in dmesg or journalctl -b seen on my system anymore.

Motherboard: ASUS PRIME X570-PRO (latest BIOS v 4021)
CPU: Ryzen 9 3950X

Other HW shouldn't matter I guess.

tags: added: verification-done-focal
removed: verification-needed-focal
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers