Decode ARM CPER records in kernel

Bug #1770244 reported by dann frazier on 2018-05-09
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Undecided
dann frazier
Bionic
Undecided
dann frazier

Bug Description

[Impact]
When an ARM-specific platform error (CPER) comes occurs, the kernel will emit an error with an opaque hex error type. The user would then need to consult the UEFI specification to decode it. It is far easier for the kernel to do the decoding itself, and just tell the user what the problem is.

[Test Case]
On a server that supports EINJ, generate a fake CE (thanks to Tyler Baicar for this example):

modprobe einj
echo 0x12345000 > /sys/kernel/debug/apei/einj/param1
echo $((-1 << 12)) > /sys/kernel/debug/apei/einj/param2
echo 5 > /sys/kernel/debug/apei/einj/param3
echo 0x3 > /sys/kernel/debug/apei/einj/flags
echo 0x1 > /sys/kernel/debug/apei/einj/error_type
echo 1 > /sys/kernel/debug/apei/einj/error_inject

In the output, look for a decoded message. Without this fix, you'll see:
[ 388.094304] {2}[Hardware Error]: error_info: 0x0000000004c6007f
[ 388.094341] {2}[Hardware Error]: physical fault address: 0x0000000012345000

But with the fix, you'll see:

[ 388.094304] {2}[Hardware Error]: error_info: 0x0000000004c6007f
[ 388.094317] {2}[Hardware Error]: transaction type: Generic
[ 388.094322] {2}[Hardware Error]: operation type: Generic read (type of instruction or data request cannot be determined)
[ 388.094326] {2}[Hardware Error]: cache level: 3
[ 388.094330] {2}[Hardware Error]: processor context not corrupted
[ 388.094333] {2}[Hardware Error]: the error has been corrected
[ 388.094337] {2}[Hardware Error]: PC is imprecise
[ 388.094341] {2}[Hardware Error]: physical fault address: 0x0000000012345000

[Fix]
These upstream fixes add ARM decoding support:

c6d8c8ef1d0d94fdae9f5d72982963db89f9cdad efi: Move ARM CPER code to new file
301f55b1a9177132d2b9ce8a90bf0ae4b37bb850 efi: Parse ARM error information value

[Regression Risk]
The code changed is specific to ARM, and has been tested there. There's a small change to arch-independent code, but it just involves renaming an array and adding the obviously correct const attribute.

CVE References

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1770244

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Changed in linux (Ubuntu Bionic):
status: New → Incomplete
dann frazier (dannf) on 2018-05-09
Changed in linux (Ubuntu Bionic):
assignee: nobody → dann frazier (dannf)
Changed in linux (Ubuntu):
assignee: nobody → dann frazier (dannf)
description: updated
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Changed in linux (Ubuntu Bionic):
status: Incomplete → Confirmed
dann frazier (dannf) on 2018-05-23
description: updated
dann frazier (dannf) on 2018-05-23
Changed in linux (Ubuntu):
status: Confirmed → In Progress
Changed in linux (Ubuntu Bionic):
status: Confirmed → In Progress
description: updated
dann frazier (dannf) on 2018-05-24
description: updated
Changed in linux (Ubuntu Bionic):
status: In Progress → Fix Committed
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
dann frazier (dannf) wrote :

Verified:

[ 330.966942] EINJ: Error INJection is initialized.
[ 350.598563] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2
[ 350.598586] {1}[Hardware Error]: It has been corrected by h/w and requires no further action
[ 350.598589] {1}[Hardware Error]: event severity: corrected
[ 350.598598] {1}[Hardware Error]: precise tstamp: 2018-06-14 22:31:22
[ 350.598602] {1}[Hardware Error]: Error 0, type: corrected
[ 350.598612] {1}[Hardware Error]: section_type: ARM processor error
[ 350.598618] {1}[Hardware Error]: MIDR: 0x00000000510fc001
[ 350.598632] {1}[Hardware Error]: Multiprocessor Affinity Register (MPIDR): 0x0000000000000201
[ 350.598640] {1}[Hardware Error]: error affinity level: 2
[ 350.598644] {1}[Hardware Error]: running state: 0x1
[ 350.598648] {1}[Hardware Error]: Power State Coordination Interface state: 0
[ 350.598654] {1}[Hardware Error]: Error info structure 0:
[ 350.598656] {1}[Hardware Error]: num errors: 2
[ 350.598661] {1}[Hardware Error]: first error captured
[ 350.598664] {1}[Hardware Error]: error_type: 0, cache error
[ 350.598668] {1}[Hardware Error]: error_info: 0x0000000004c6007f
[ 350.598674] {1}[Hardware Error]: transaction type: Generic
[ 350.598677] {1}[Hardware Error]: operation type: Generic read (type of instruction or data request cannot be determined)
[ 350.598680] {1}[Hardware Error]: cache level: 3
[ 350.598684] {1}[Hardware Error]: processor context not corrupted
[ 350.598688] {1}[Hardware Error]: the error has been corrected
[ 350.598691] {1}[Hardware Error]: PC is imprecise
[ 350.598695] {1}[Hardware Error]: physical fault address: 0x0000000012345000

tags: added: verification-done-bionic
removed: verification-needed-bionic
Launchpad Janitor (janitor) wrote :
Download full text (49.5 KiB)

This bug was fixed in the package linux - 4.15.0-24.26

---------------
linux (4.15.0-24.26) bionic; urgency=medium

  * linux: 4.15.0-24.26 -proposed tracker (LP: #1776338)

  * Bionic update: upstream stable patchset 2018-06-06 (LP: #1775483)
    - drm: bridge: dw-hdmi: Fix overflow workaround for Amlogic Meson GX SoCs
    - i40e: Fix attach VF to VM issue
    - tpm: cmd_ready command can be issued only after granting locality
    - tpm: tpm-interface: fix tpm_transmit/_cmd kdoc
    - tpm: add retry logic
    - Revert "ath10k: send (re)assoc peer command when NSS changed"
    - bonding: do not set slave_dev npinfo before slave_enable_netpoll in
      bond_enslave
    - ipv6: add RTA_TABLE and RTA_PREFSRC to rtm_ipv6_policy
    - ipv6: sr: fix NULL pointer dereference in seg6_do_srh_encap()- v4 pkts
    - KEYS: DNS: limit the length of option strings
    - l2tp: check sockaddr length in pppol2tp_connect()
    - net: validate attribute sizes in neigh_dump_table()
    - llc: delete timers synchronously in llc_sk_free()
    - tcp: don't read out-of-bounds opsize
    - net: af_packet: fix race in PACKET_{R|T}X_RING
    - tcp: md5: reject TCP_MD5SIG or TCP_MD5SIG_EXT on established sockets
    - net: fix deadlock while clearing neighbor proxy table
    - team: avoid adding twice the same option to the event list
    - net/smc: fix shutdown in state SMC_LISTEN
    - team: fix netconsole setup over team
    - packet: fix bitfield update race
    - tipc: add policy for TIPC_NLA_NET_ADDR
    - pppoe: check sockaddr length in pppoe_connect()
    - vlan: Fix reading memory beyond skb->tail in skb_vlan_tagged_multi
    - amd-xgbe: Add pre/post auto-negotiation phy hooks
    - sctp: do not check port in sctp_inet6_cmp_addr
    - amd-xgbe: Improve KR auto-negotiation and training
    - strparser: Do not call mod_delayed_work with a timeout of LONG_MAX
    - amd-xgbe: Only use the SFP supported transceiver signals
    - strparser: Fix incorrect strp->need_bytes value.
    - net: sched: ife: signal not finding metaid
    - tcp: clear tp->packets_out when purging write queue
    - net: sched: ife: handle malformed tlv length
    - net: sched: ife: check on metadata length
    - llc: hold llc_sap before release_sock()
    - llc: fix NULL pointer deref for SOCK_ZAPPED
    - net: ethernet: ti: cpsw: fix tx vlan priority mapping
    - virtio_net: split out ctrl buffer
    - virtio_net: fix adding vids on big-endian
    - KVM: s390: force bp isolation for VSIE
    - s390: correct module section names for expoline code revert
    - microblaze: Setup dependencies for ASM optimized lib functions
    - commoncap: Handle memory allocation failure.
    - scsi: mptsas: Disable WRITE SAME
    - cdrom: information leak in cdrom_ioctl_media_changed()
    - m68k/mac: Don't remap SWIM MMIO region
    - block/swim: Check drive type
    - block/swim: Don't log an error message for an invalid ioctl
    - block/swim: Remove extra put_disk() call from error path
    - block/swim: Rename macros to avoid inconsistent inverted logic
    - block/swim: Select appropriate drive on device open
    - block/swim: Fix array bounds check
    - block/swim: Fix IO error at end of medium
    -...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :
Download full text (4.1 KiB)

This bug was fixed in the package linux - 4.15.0-29.31

---------------
linux (4.15.0-29.31) bionic; urgency=medium

  * linux: 4.15.0-29.31 -proposed tracker (LP: #1782173)

  * [SRU Bionic][Cosmic] kernel panic in ipmi_ssif at msg_done_handler
    (LP: #1777716)
    - ipmi_ssif: Fix kernel panic at msg_done_handler

  * Update to ocxl driver for 18.04.1 (LP: #1775786)
    - misc: ocxl: use put_device() instead of device_unregister()
    - powerpc: Add TIDR CPU feature for POWER9
    - powerpc: Use TIDR CPU feature to control TIDR allocation
    - powerpc: use task_pid_nr() for TID allocation
    - ocxl: Rename pnv_ocxl_spa_remove_pe to clarify it's action
    - ocxl: Expose the thread_id needed for wait on POWER9
    - ocxl: Add an IOCTL so userspace knows what OCXL features are available
    - ocxl: Document new OCXL IOCTLs
    - ocxl: Fix missing unlock on error in afu_ioctl_enable_p9_wait()

  * Critical upstream bugfix missing in Ubuntu 18.04 - frequent Xorg crash after
    suspend (LP: #1776887)
    - ocxl: Document the OCXL_IOCTL_GET_METADATA IOCTL

  * Hard LOCKUP observed on stressing Ubuntu 18 04 (LP: #1777194)
    - powerpc: use NMI IPI for smp_send_stop
    - powerpc: Fix smp_send_stop NMI IPI handling

  * IPL: ppc64_cpu --frequency hang with INFO: rcu_sched detected stalls on
    CPUs/tasks on w34 and wsbmc016 with 920.1714.20170330n (LP: #1773964)
    - rtc: opal: Fix OPAL RTC driver OPAL_BUSY loops

  * [Regression] EXT4-fs error (device sda2): ext4_validate_block_bitmap:383:
    comm stress-ng: bg 4705: bad block bitmap checksum (LP: #1781709)
    - SAUCE: Revert "UBUNTU: SAUCE: ext4: fix ext4_validate_inode_bitmap: comm
      stress-ng: Corrupt inode bitmap"
    - SAUCE: ext4: check for allocation block validity with block group locked

linux (4.15.0-28.30) bionic; urgency=medium

  * linux: 4.15.0-28.30 -proposed tracker (LP: #1781433)

  * Cannot set MTU higher than 1500 in Xen instance (LP: #1781413)
    - xen-netfront: Fix mismatched rtnl_unlock
    - xen-netfront: Update features after registering netdev

linux (4.15.0-27.29) bionic; urgency=medium

  * linux: 4.15.0-27.29 -proposed tracker (LP: #1781062)

  * [Regression] EXT4-fs error (device sda1): ext4_validate_inode_bitmap:99:
    comm stress-ng: Corrupt inode bitmap (LP: #1780137)
    - SAUCE: ext4: fix ext4_validate_inode_bitmap: comm stress-ng: Corrupt inode
      bitmap

linux (4.15.0-26.28) bionic; urgency=medium

  * linux: 4.15.0-26.28 -proposed tracker (LP: #1780112)

  * failure to boot with linux-image-4.15.0-24-generic (LP: #1779827) // Cloud-
    init causes potentially huge boot delays with 4.15 kernels (LP: #1780062)
    - random: Make getrandom() ready earlier

linux (4.15.0-25.27) bionic; urgency=medium

  * linux: 4.15.0-25.27 -proposed tracker (LP: #1779354)

  * hisi_sas_v3_hw: internal task abort: timeout and not done. (LP: #1777736)
    - scsi: hisi_sas: Update a couple of register settings for v3 hw

  * hisi_sas: Add missing PHY spinlock init (LP: #1777734)
    - scsi: hisi_sas: Add missing PHY spinlock init

  * hisi_sas: improve read performance by pre-allocating slot DMA buffers
    (LP: #1777727)
    - scsi: hisi_sas: use dma_zalloc_cohe...

Read more...

Changed in linux (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers