powerpc/livepatch: Implement reliable stack tracing for the consistency model

Bug #1771844 reported by bugproxy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
Fix Released
High
Canonical Kernel Team
linux (Ubuntu)
Fix Released
High
Joseph Salisbury
Bionic
Fix Released
High
Joseph Salisbury
Cosmic
Fix Released
High
Joseph Salisbury

Bug Description

== SRU Justification ==
Livepatch has a consistency model which is a hybrid of kGraft and kpatch:
it uses kGraft's per-task consistency and syscall barrier switching
combined with kpatch's stack trace switching. The current approach is
stack checking of sleeping tasks. If no affected functions are on the
stack of a given task, the task is patched. In most cases this will patch
most or all of the tasks on the first try. Otherwise, it'll keep trying
periodically. This patch implements the reliable stack tracing for
consistency model a.k.a HAVE_RELIABLE_STACKTRACE.

This will help in switching livepatching implementation to basic per-task
consistency model. It is the foundation, which will help us enable
security patches changing function or data semantics. This is the biggest
remaining piece needed on ppc64le to make livepatch more generally useful.

== Fix ==
df78d3f61480 ("powerpc/livepatch: Implement reliable stack tracing for the consistency model")

== Regression Potential ==
Low. Limited to powerpc.

== Test Case ==
A test kernel was built with this patch and tested by the original bug reporter.
The bug reporter states the test kernel resolved the bug.

https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git/commit/?id=df78d3f6148092d33a9a24c7a9cfac

CVE References

bugproxy (bugproxy)
tags: added: architecture-ppc64le bugnameltc-167879 severity-critical targetmilestone-inin1804
Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → linux (Ubuntu)
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
status: New → Triaged
tags: added: triage-g
Changed in ubuntu-power-systems:
importance: Undecided → High
assignee: nobody → Canonical Kernel Team (canonical-kernel-team)
bugproxy (bugproxy)
tags: added: targetmilestone-inin18041
removed: targetmilestone-inin1804
bugproxy (bugproxy)
tags: added: targetmilestone-inin1804
removed: targetmilestone-inin18041
Changed in linux (Ubuntu):
importance: Undecided → High
status: New → Triaged
Changed in linux (Ubuntu Bionic):
status: New → In Progress
Changed in linux (Ubuntu Cosmic):
status: Triaged → In Progress
Changed in linux (Ubuntu Bionic):
assignee: nobody → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Cosmic):
assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → Joseph Salisbury (jsalisbury)
Changed in linux (Ubuntu Bionic):
importance: Undecided → High
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

I built a test kernel with commit df78d3f6148092d33a9a24c7a9cfac3d0220b484. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1771844

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-image-unsigned, linux-modules and linux-modules-extra .deb packages.

Thanks in advance!

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2018-05-22 00:13 EDT-------
(In reply to comment #5)
> I built a test kernel with commit df78d3f6148092d33a9a24c7a9cfac3d0220b484.
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1771844
>
> Can you test this kernel and see if it resolves this bug?
>
> Note about installing test kernels:
> ? If the test kernel is prior to 4.15(Bionic) you need to install the
> linux-image and linux-image-extra .deb packages.
> ? If the test kernel is 4.15(Bionic) or newer, you need to install the
> linux-image-unsigned, linux-modules and linux-modules-extra .deb packages.
>
> Thanks in advance!

Thanks for building the kernel with the patch. Can you share the kernel sources, so that can I build the livepatch modules based on the sources. Standard .config, does not build the samples/livepatch to do the basic testing.
cat /boot/config-4.15.0-20-generic |grep -i livepatch
CONFIG_HAVE_LIVEPATCH=y
CONFIG_LIVEPATCH=y

I would also try to build livepatch modules other than the sample modules in the kernel sources to verify save_stack_trace functionality.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

I built a new test kernel and included the source package.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1771844

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-05-23 06:27 EDT-------
(In reply to comment #7)
> I built a new test kernel and included the source package.
>
> The test kernel can be downloaded from:
> http://kernel.ubuntu.com/~jsalisbury/lp1771844

Thanks for sharing the kernel sources. I was able to verify the patch with livepatch-sample.ko :
[ 4772.567633] livepatch_sample: tainting kernel with TAINT_LIVEPATCH
[ 4772.569747] livepatch_sample: module verification failed: signature and/or required key missing - tainting kernel
[ 4772.579440] livepatch: enabling patch 'livepatch_sample'
[ 4772.580686] livepatch: 'livepatch_sample': starting patching transition
[ 4777.032952] livepatch: 'livepatch_sample': patching complete
[ 4800.732461] livepatch: 'livepatch_sample': starting unpatching transition
[ 4804.969121] livepatch: 'livepatch_sample': unpatching complete

transition (un)patching works as expected.

Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
status: Triaged → In Progress
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :
description: updated
Changed in linux (Ubuntu Bionic):
status: In Progress → Fix Committed
Manoj Iyer (manjo)
Changed in ubuntu-power-systems:
status: In Progress → Fix Committed
Changed in linux (Ubuntu Cosmic):
status: In Progress → Fix Committed
Revision history for this message
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-06-15 04:33 EDT-------
The proposed Bionic kernel enables the 'Implement reliable stack tracing for the consistency model' patch.

In brief:
------------
After upgrading the system by enabling bionic-proposed to sources.list:
deb http://archive.ubuntu.com/ubuntu/ bionic-proposed restricted main multiverse universe

system booted into kernel version 4.15.0-24-generic, using 4.15.0-24-generic sources (apt source linux) to build livepatch modules:
# insmod ./livepatch-0001-mm-thp-Do-not-make-page-table-dir.ko
# dmesg
.....
[ 1827.037453] livepatch_0001_mm_thp_Do_not_make_page_table_dir: loading out-of-tree module taints kernel.
[ 1827.037458] livepatch_0001_mm_thp_Do_not_make_page_table_dir: tainting kernel with TAINT_LIVEPATCH
[ 1827.043106] livepatch_0001_mm_thp_Do_not_make_page_table_dir: module verification failed: signature and/or required key missing - tainting kernel
[ 1827.176933] livepatch: enabling patch 'livepatch_0001_mm_thp_Do_not_make_page_table_dir'
[ 1827.179784] livepatch: 'livepatch_0001_mm_thp_Do_not_make_page_table_dir': starting patching transition
[ 1827.179785] livepatch: 'livepatch_0001_mm_thp_Do_not_make_page_table_dir': patching complete
[ 2526.521348] livepatch: 'livepatch_0001_mm_thp_Do_not_make_page_table_dir': starting unpatching transition
[ 2526.525554] livepatch: 'livepatch_0001_mm_thp_Do_not_make_page_table_dir': unpatching complete

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (49.5 KiB)

This bug was fixed in the package linux - 4.15.0-24.26

---------------
linux (4.15.0-24.26) bionic; urgency=medium

  * linux: 4.15.0-24.26 -proposed tracker (LP: #1776338)

  * Bionic update: upstream stable patchset 2018-06-06 (LP: #1775483)
    - drm: bridge: dw-hdmi: Fix overflow workaround for Amlogic Meson GX SoCs
    - i40e: Fix attach VF to VM issue
    - tpm: cmd_ready command can be issued only after granting locality
    - tpm: tpm-interface: fix tpm_transmit/_cmd kdoc
    - tpm: add retry logic
    - Revert "ath10k: send (re)assoc peer command when NSS changed"
    - bonding: do not set slave_dev npinfo before slave_enable_netpoll in
      bond_enslave
    - ipv6: add RTA_TABLE and RTA_PREFSRC to rtm_ipv6_policy
    - ipv6: sr: fix NULL pointer dereference in seg6_do_srh_encap()- v4 pkts
    - KEYS: DNS: limit the length of option strings
    - l2tp: check sockaddr length in pppol2tp_connect()
    - net: validate attribute sizes in neigh_dump_table()
    - llc: delete timers synchronously in llc_sk_free()
    - tcp: don't read out-of-bounds opsize
    - net: af_packet: fix race in PACKET_{R|T}X_RING
    - tcp: md5: reject TCP_MD5SIG or TCP_MD5SIG_EXT on established sockets
    - net: fix deadlock while clearing neighbor proxy table
    - team: avoid adding twice the same option to the event list
    - net/smc: fix shutdown in state SMC_LISTEN
    - team: fix netconsole setup over team
    - packet: fix bitfield update race
    - tipc: add policy for TIPC_NLA_NET_ADDR
    - pppoe: check sockaddr length in pppoe_connect()
    - vlan: Fix reading memory beyond skb->tail in skb_vlan_tagged_multi
    - amd-xgbe: Add pre/post auto-negotiation phy hooks
    - sctp: do not check port in sctp_inet6_cmp_addr
    - amd-xgbe: Improve KR auto-negotiation and training
    - strparser: Do not call mod_delayed_work with a timeout of LONG_MAX
    - amd-xgbe: Only use the SFP supported transceiver signals
    - strparser: Fix incorrect strp->need_bytes value.
    - net: sched: ife: signal not finding metaid
    - tcp: clear tp->packets_out when purging write queue
    - net: sched: ife: handle malformed tlv length
    - net: sched: ife: check on metadata length
    - llc: hold llc_sap before release_sock()
    - llc: fix NULL pointer deref for SOCK_ZAPPED
    - net: ethernet: ti: cpsw: fix tx vlan priority mapping
    - virtio_net: split out ctrl buffer
    - virtio_net: fix adding vids on big-endian
    - KVM: s390: force bp isolation for VSIE
    - s390: correct module section names for expoline code revert
    - microblaze: Setup dependencies for ASM optimized lib functions
    - commoncap: Handle memory allocation failure.
    - scsi: mptsas: Disable WRITE SAME
    - cdrom: information leak in cdrom_ioctl_media_changed()
    - m68k/mac: Don't remap SWIM MMIO region
    - block/swim: Check drive type
    - block/swim: Don't log an error message for an invalid ioctl
    - block/swim: Remove extra put_disk() call from error path
    - block/swim: Rename macros to avoid inconsistent inverted logic
    - block/swim: Select appropriate drive on device open
    - block/swim: Fix array bounds check
    - block/swim: Fix IO error at end of medium
    -...

Changed in linux (Ubuntu Bionic):
status: Fix Committed → Fix Released
Changed in linux (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Changed in ubuntu-power-systems:
status: Fix Committed → Fix Released
Brad Figg (brad-figg)
tags: added: cscc
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.