[Bug][CLX]assertion failure with util_range_rw using libpmemlog, possible kernel DAX bug

Bug #1789146 reported by quanxian on 2018-08-27
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
intel
Medium
Unassigned
linux (Ubuntu)
Medium
Joseph Salisbury

Bug Description

Description:
On Mon, Apr 16, 2018 at 8:20 AM, Czurylo, Krzysztof
<email address hidden> wrote:
>
> I suspect the problem is caused by a bug in the kernel.
>
> I did a few experiments and it looks like the issue occurs only if the
> filesystem is mounted with "-o dax". I can reproduce is both for xfs
> and ext4, so it's not FS-specific, but rather DAX-specific. It also
> reproduces on an emulated PMEM - no need to use real AEP DIMMs.
>
> Using the latest kernel (4.16.0) does not help.
>
> What happens:
>
> In debug version of libpmemlog (but also libpmemblk), the entire pool
> is by default write-protected with mprotect(..., PROT_READ).
>
> When the program attempts to write some data to the pool (i.e.
> pmemlog_append, pmemblk_write, ...), the library unprotects the pages
> to be modified (usually just one or two pages) and once the data is
> stored, the pages are protected again.
>
> Inside the kernel, mprotect splits the memory region associated with
> the pool into 3 regions: the read-only head and tail + one r/w page in
> the middle.
>
> The problem is that after the last step, the memory region associated
> with the modified page is not merged with the adjacent regions having
> the same protection flags (ro) to form one big read-only region again.
> This leads to the situation where we have thousands of 4K memory
> mappings per process that are tracked by the kernel separately. When
> the number of maps exceeds the limit (default is 65536 - see:
> /proc/sys/vm/max_map_count), mprotect fails with ENOMEM, which aborts
> the program.

Commitid: e1fb4a0864958fac2fb1b23f9f4562a9f90e3e8f
dax: remove VM_MIXEDMAP for fsdax and device dax

Target Kernel: 4.19

Target Release: 18.10

CVE References

quanxian (quanxian-wang) wrote :

this patch is not in 4.18.

If possible, please cherry pick it into Ubuntu 18.10. Thanks

quanxian (quanxian-wang) wrote :

clx platform is not PV, just keep it private. Thanks

quanxian (quanxian-wang) on 2018-08-27
information type: Public → Private
quanxian (quanxian-wang) wrote :

need backporting

Changed in intel:
status: New → Triaged
importance: Undecided → Medium
Changed in ubuntu:
importance: Undecided → Medium
status: New → Triaged
affects: ubuntu → linux (Ubuntu)
Joseph Salisbury (jsalisbury) wrote :

I built a test kernel with commit e1fb4a0864958fac2fb1b23f9f4562a9f90e3e8f. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1789146

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.

Thanks in advance!

Changed in linux (Ubuntu):
assignee: nobody → Joseph Salisbury (jsalisbury)
status: Triaged → In Progress
quanxian (quanxian-wang) wrote :

I am finding the test case from upstream to have a try.

quanxian (quanxian-wang) wrote :

hi, Joseph

Our test has tested in 4.16(no patch) and 4.19(with the patch), it works.

But with your build image, we don't find the bug is fixed. Would you like to double check if the patch has been integrated your kernel. Thanks

Quanxian

Changed in linux (Ubuntu):
status: In Progress → Fix Committed
information type: Private → Public
Joseph Salisbury (jsalisbury) wrote :

I built a 18.10 test kernel using the latest master-next branch of the repo. This branch has the patch applied.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1789146

Can you test this kernel and see if it resolves this bug?

quanxian (quanxian-wang) wrote :

hi, Joseph

I have built a kernel by myself clone from below link. The version is 4.18.

git://kernel.ubuntu.com/kernel-ppa/mirror/ubuntu-cosmic.git
commit d4b160782ac74f5301651346495903a30cf752d3 (HEAD -> master, tag: Ubuntu-4.18.0-7.8, origin/master, origin/HEAD)
Author: Seth Forshee <email address hidden>
Date: Tue Aug 28 11:09:06 2018 -0500

    UBUNTU: Ubuntu-4.18.0-7.8

    Signed-off-by: Seth Forshee <email address hidden>

in this branch, there is no patch. Test failed.
I apply the new patch, the test works.

Therefore for 4.18, it will be fine with patch.

For your new kernel, I will have a try, will let you know the result.

quanxian (quanxian-wang) wrote :

we have verified your kernel package, it works. Thanks

Launchpad Janitor (janitor) wrote :
Download full text (60.2 KiB)

This bug was fixed in the package linux - 4.18.0-9.10

---------------
linux (4.18.0-9.10) cosmic; urgency=medium

  * linux: 4.18.0-9.10 -proposed tracker (LP: #1796346)

  * Cosmic update: v4.18.12 upstream stable release (LP: #1796139)
    - crypto: skcipher - Fix -Wstringop-truncation warnings
    - iio: adc: ina2xx: avoid kthread_stop() with stale task_struct
    - tsl2550: fix lux1_input error in low light
    - misc: ibmvmc: Use GFP_ATOMIC under spin lock
    - vmci: type promotion bug in qp_host_get_user_memory()
    - siox: don't create a thread without starting it
    - x86/numa_emulation: Fix emulated-to-physical node mapping
    - staging: rts5208: fix missing error check on call to rtsx_write_register
    - power: supply: axp288_charger: Fix initial constant_charge_current value
    - misc: sram: enable clock before registering regions
    - serial: sh-sci: Stop RX FIFO timer during port shutdown
    - uwb: hwa-rc: fix memory leak at probe
    - power: vexpress: fix corruption in notifier registration
    - iommu/amd: make sure TLB to be flushed before IOVA freed
    - Bluetooth: Add a new Realtek 8723DE ID 0bda:b009
    - USB: serial: kobil_sct: fix modem-status error handling
    - 6lowpan: iphc: reset mac_header after decompress to fix panic
    - iommu/msm: Don't call iommu_device_{,un}link from atomic context
    - s390/mm: correct allocate_pgste proc_handler callback
    - power: remove possible deadlock when unregistering power_supply
    - drm/amd/display/dc/dce: Fix multiple potential integer overflows
    - drm/amd/display: fix use of uninitialized memory
    - md-cluster: clear another node's suspend_area after the copy is finished
    - cxgb4: Fix the condition to check if the card is T5
    - RDMA/bnxt_re: Fix a couple off by one bugs
    - RDMA/i40w: Hold read semaphore while looking after VMA
    - RDMA/bnxt_re: Fix a bunch of off by one bugs in qplib_fp.c
    - IB/core: type promotion bug in rdma_rw_init_one_mr()
    - media: exynos4-is: Prevent NULL pointer dereference in __isp_video_try_fmt()
    - IB/mlx4: Test port number before querying type.
    - powerpc/kdump: Handle crashkernel memory reservation failure
    - media: fsl-viu: fix error handling in viu_of_probe()
    - vhost_net: Avoid tx vring kicks during busyloop
    - media: staging/imx: fill vb2_v4l2_buffer field entry
    - IB/mlx5: Fix GRE flow specification
    - include/rdma/opa_addr.h: Fix an endianness issue
    - x86/tsc: Add missing header to tsc_msr.c
    - ARM: hwmod: RTC: Don't assume lock/unlock will be called with irq enabled
    - x86/entry/64: Add two more instruction suffixes
    - ARM: dts: ls1021a: Add missing cooling device properties for CPUs
    - scsi: target/iscsi: Make iscsit_ta_authentication() respect the output
      buffer size
    - thermal: i.MX: Allow thermal probe to fail gracefully in case of bad
      calibration.
    - scsi: klist: Make it safe to use klists in atomic context
    - scsi: ibmvscsi: Improve strings handling
    - scsi: target: Avoid that EXTENDED COPY commands trigger lock inversion
    - usb: wusbcore: security: cast sizeof to int for comparison
    - ath10k: sdio: use same endpoint id for all packets...

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
quanxian (quanxian-wang) on 2018-10-26
Changed in intel:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers