kpartx 0.7.4-2ubuntu1 in bionic does not remove loop devices

Bug #1747044 reported by Francis Ginther
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
multipath-tools (Ubuntu)
Fix Released
Critical
Julian Andres Klode

Bug Description

We're seeing a problem now when running livecd-rootfs in bionic with kpartx from 0.7.4-2ubuntu1. The issue appears to be the result in a change in kpartx, as a result, the /dev/loop devices are not getting deleted. The best data I have from this so far is a comparison between old and current build logs. The old logs showed:

++ kpartx -v -d binary/boot/disk-uefi.ext4
+ kpartx_stdout='del devmap : loop0p15
del devmap : loop0p14
del devmap : loop0p1
loop deleted : /dev/loop0'
+ echo 'del devmap : loop0p15
del devmap : loop0p14
del devmap : loop0p1
loop deleted : /dev/loop0'

Thew new logs show:

++ kpartx -v -d binary/boot/disk-uefi.ext4
+ kpartx_stdout=
+ echo ''

The code being run is from the 'functions' in lp:livecd-rootfs. In the current runs, kpartx_stdout is an empty string:

clean_loops() {
    local kpartx_ret
    local kpartx_stdout

    if [ -n "${backing_img}" ]; then
        # sync before removing loop to avoid "Device or resource busy" errors
        sync
        kpartx_ret=""
        kpartx_stdout=$(kpartx -v -d "${backing_img}") || kpartx_ret=$?
        echo "$kpartx_stdout"
        if [ -n "$kpartx_ret" ]; then
            if echo "$kpartx_stdout" | grep -q "loop deleted"; then
                echo "Suppressing kpartx returning error (#860894)"
            else
                exit $kpartx_ret
            fi
        fi
        unset backing_img
    fi

    if [ -z "${rootfs_dev_mapper}" ]; then
        return 0
    fi

    unset loop_device
    unset loop_raw
    unset rootfs_dev_mapper
}

I am doing some further debugging on this and am trying to reproduce this outside of launchpad. I am not able to post a full build log. But I can probably post a longer 'sanitized' bit.

summary: - kpartx output has changed in 0.7.4-2ubuntu1 package
+ kpartx 0.7.4-2ubuntu1 in bionic can't remove loop devices
summary: - kpartx 0.7.4-2ubuntu1 in bionic can't remove loop devices
+ kpartx 0.7.4-2ubuntu1 in bionic does not remove loop devices
description: updated
Revision history for this message
Francis Ginther (fginther) wrote :

I have a more concise method for showing this problem. When I try to mount the partitions in an img file with kpartx 0.7.4-2ubuntu1, it silently fails:

# First, fetch an image
$ wget https://cloud-images.ubuntu.com/bionic/20180124/bionic-server-cloudimg-amd64.img
...
$ sudo kpartx -v -l bionic-server-cloudimg-amd64.img
loop deleted : /dev/loop0
$ sudo kpartx -v -s -a bionic-server-cloudimg-amd64.img
$ ll /dev/mapper/
total 0
drwxr-xr-x 2 root root 60 Feb 2 20:03 ./
drwxr-xr-x 16 root root 3580 Feb 2 20:03 ../
crw------- 1 root root 10, 236 Feb 2 20:03 control

The same works with 0.6.4-5ubuntu1 (and it can even delete the loop device):

$ sudo kpartx -v -l bionic-server-cloudimg-amd64.img
loop0p1 : 0 598525 /dev/loop0 3
loop deleted : /dev/loop0
$ sudo kpartx -v -s -a bionic-server-cloudimg-amd64.img
add map loop0p1 (253:0): 0 598525 linear 7:0 3
$ ll /dev/mapper
total 0
drwxr-xr-x 2 root root 80 Feb 2 20:08 ./
drwxr-xr-x 16 root root 3600 Feb 2 20:08 ../
crw------- 1 root root 10, 236 Feb 2 20:04 control
lrwxrwxrwx 1 root root 7 Feb 2 20:08 loop0p1 -> ../dm-0

$ sudo kpartx -v -d bionic-server-cloudimg-amd64.img
del devmap : loop0p1
loop deleted : /dev/loop0
$ ll /dev/mapper
total 0
drwxr-xr-x 2 root root 60 Feb 2 20:13 ./
drwxr-xr-x 16 root root 3580 Feb 2 20:13 ../
crw------- 1 root root 10, 236 Feb 2 20:04 control

Revision history for this message
Francis Ginther (fginther) wrote :

The earlier steps to reproduce were wrong. I failed to convert the img file to a raw file first. So I should have:

# Get a cloud image and convert it to raw format:
$ wget https://cloud-images.ubuntu.com/bionic/20180124/bionic-server-cloudimg-amd64.img
$ qemu-img convert -O raw bionic-server-cloudimg-amd64.img bionic-server-cloudimg-amd64.raw

# Now kpartx will attach the loop device:
$ sudo kpartx -v -l bionic-server-cloudimg-amd64.raw
loop3p1 : 0 4384735 /dev/loop3 227328
loop3p14 : 0 8192 /dev/loop3 2048
loop3p15 : 0 217088 /dev/loop3 10240
loop deleted : /dev/loop3
$ sudo kpartx -v -s -a bionic-server-cloudimg-amd64.raw
add map loop3p1 (253:0): 0 4384735 linear 7:3 227328
add map loop3p14 (253:1): 0 8192 linear 7:3 2048
add map loop3p15 (253:2): 0 217088 linear 7:3 10240

# But removing does nothing:
$ sudo kpartx -v -d bionic-server-cloudimg-amd64.raw
$ ll /dev/mapper
total 0
drwxr-xr-x 2 root root 120 Feb 2 20:43 ./
drwxr-xr-x 16 root root 3640 Feb 2 20:43 ../
crw------- 1 root root 10, 236 Feb 2 20:03 control
lrwxrwxrwx 1 root root 7 Feb 2 20:43 loop3p1 -> ../dm-0
lrwxrwxrwx 1 root root 7 Feb 2 20:43 loop3p14 -> ../dm-1
lrwxrwxrwx 1 root root 7 Feb 2 20:43 loop3p15 -> ../dm-2

Steve Langasek (vorlon)
Changed in multipath-tools (Ubuntu):
status: New → Confirmed
importance: Undecided → Critical
assignee: nobody → Julian Andres Klode (juliank)
Revision history for this message
Julian Andres Klode (juliank) wrote :

Surprisingly enough kpartx -d /dev/loop<foo> works.

Changed in multipath-tools (Ubuntu):
status: Confirmed → Won't Fix
status: Won't Fix → In Progress
Revision history for this message
Julian Andres Klode (juliank) wrote :

Uploaded

multipath-tools (0.7.4-2ubuntu2) bionic; urgency=medium

  * d/p/kpartx-Improve-finding-loopback-device-by-file.patch: Improve
    finding loopback devices (LP: #1747044)

Date: Mon, 05 Feb 2018 10:06:19 +0100
Changed-By: Julian Andres Klode <email address hidden>
Maintainer: Ubuntu Developers <email address hidden>
https://launchpad.net/ubuntu/+source/multipath-tools/0.7.4-2ubuntu2

and submitted a patch upstream in https://www.redhat.com/archives/dm-devel/2018-February/msg00019.html

Changed in multipath-tools (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package multipath-tools - 0.7.4-2ubuntu3

---------------
multipath-tools (0.7.4-2ubuntu3) bionic; urgency=medium

  * d/p/kpartx-print-loop-deleted-to-stdout-not-stderr.patch: Print loop
    deleted message to stdout again, not stderr; to fix autopkgtest

 -- Julian Andres Klode <email address hidden> Mon, 05 Feb 2018 11:46:37 +0100

Changed in multipath-tools (Ubuntu):
status: Fix Committed → Fix Released
tags: added: id-5a74d37c8d7c3685c37e0584
tags: added: id-5a73df16c347ab07bfffa3d7
Revision history for this message
Utkarsh Gupta (utkarsh) wrote (last edit ):

Hello,

Upstream sanctioned an alternate patch (cf: https://github.com/opensvc/multipath-tools/commit/95c4189b683c543ac291628df8143a93373de17b) which tries to solve the same problem, as discussed on the upstream mailing list (cf: thread here: https://listman.redhat.com/archives/dm-devel/2018-February/msg00019.html).

I am inclined towards giving this a try? How do you feel about this? Is this something we can adapt to with 0.8.6 (which is released upstream but yet to be uploaded to the archive)?

Do note that this patch is already included in the version which is in the archive. So what I am proposing is that should we rely on that and remove the debian/patches/kpartx_more_loopback_fixes.patch instead?

Let me know what you think? TIA! \o/

Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

As discussed on the MP for the latest multipath-tools merge at https://code.launchpad.net/~athos-ribeiro/ubuntu/+source/multipath-tools/+git/multipath-tools/+merge/415814, we are keeping d/p/kpartx-Improve-finding-loopback-device-by-file.patch for now and assessing removal on https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1961633.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.