mount-image-callback cannot mount partitioned disk image

Bug #1628336 reported by Scott Moser on 2016-09-28
This bug affects 2 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Stefan Bader
Stefan Bader

Bug Description

This fails on 4.8.0-17-generic and passes on 4.4.0-9136-generic

$ sudo apt-get update
$ sudo apt-get install -qy cloud-image-utils
$ wget
$ cp xenial-server-cloudimg-amd64-disk1.img disk.img
$ sudo mount-image-callback xenial-server-cloudimg-amd64-disk1.img -v ls _MOUNTPOINT_
replaced string _MOUNTPOINT_ in arguments arg 1
waiting on pidfile for /dev/nbd0 in /sys/block/nbd0/pid
connected xenial-server-cloudimg-amd64-disk1.img (qcow2) to /dev/nbd0. waiting for device.
partitioned disk.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
waiting for /dev/nbd0p1 part=1 to be ready.
gave up on waiting for /dev/nbd0p1
$ echo $?

On an identical system other than 'apt-get install linux-virtual' to get the new kernel, you see:

$ uname -r

$ sudo mount-image-callback disk.img -v ls _MOUNTPOINT_
replaced string _MOUNTPOINT_ in arguments arg 1
waiting on pidfile for /dev/nbd0 in /sys/block/nbd0/pid
connected disk.img (qcow2) to /dev/nbd0. waiting for device.
partitioned disk.
mounted /dev/nbd0p1 via qemu-nbd /dev/nbd0
invoking: MOUNTPOINT=/tmp/mount-image-callback.Y8D4lk/mp ls /tmp/mount-image-callback.Y8D4lk/mp
bin etc lib media proc sbin sys var
boot home lib64 mnt root snap tmp vmlinuz
dev initrd.img lost+found opt run srv usr
cmd returned 0. unmounting /tmp/mount-image-callback.Y8D4lk/mp

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: linux-image-4.8.0-17-generic 4.8.0-17.19
ProcVersionSignature: User Name 4.8.0-17.19-generic 4.8.0-rc7
Uname: Linux 4.8.0-17-generic x86_64
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 27 19:04 seq
 crw-rw---- 1 root audio 116, 33 Sep 27 19:04 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.3-0ubuntu7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Wed Sep 28 00:58:49 2016
Ec2AMI: ami-00000438
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova

 PATH=(custom, no user)

ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.8.0-17-generic root=UUID=072136f3-5666-4381-83bd-8d7175059be3 ro console=tty1 console=ttyS0
 linux-restricted-modules-4.8.0-17-generic N/A
 linux-backports-modules-4.8.0-17-generic N/A
 linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install) 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: Ubuntu-1.8.2-1ubuntu1~cloud0
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-vivid
dmi.modalias: dmi:bvnSeaBIOS:bvrUbuntu-1.8.2-1ubuntu1~cloud0:bd04/01/2014:svnOpenStackFoundation:pnOpenStackNova:pvr12.0.3:cvnQEMU:ct1:cvrpc-i440fx-vivid: OpenStack Nova
dmi.product.version: 12.0.3
dmi.sys.vendor: OpenStack Foundation

Scott Moser (smoser) wrote :
Scott Moser (smoser) wrote :

Attaching a script that shows the failure without downloading an image.

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
tags: added: kernel-4.8
Stefan Bader (smb) wrote :

I can recreate the issue following the steps in the initial description. Interestingly there are some (rare) cases where things works on 4.8 as well. There is a difference in udev messages:

Not working:
KERNEL[3512.687794] change /devices/virtual/block/loop0 (block)
KERNEL[3512.704230] change /devices/virtual/block/loop0 (block)
KERNEL[3512.705301] change /devices/virtual/block/loop0 (block)
UDEV [3512.732209] change /devices/virtual/block/loop0 (block)
UDEV [3512.748186] change /devices/virtual/block/loop0 (block)
KERNEL[3512.752751] change /devices/virtual/block/nbd0 (block)
UDEV [3512.764683] change /devices/virtual/block/nbd0 (block)
UDEV [3512.772212] change /devices/virtual/block/loop0 (block)
KERNEL[3522.912384] change /devices/virtual/block/nbd0 (block)
UDEV [3522.919200] change /devices/virtual/block/nbd0 (block)

KERNEL[3528.095845] change /devices/virtual/block/loop0 (block)
KERNEL[3528.124184] change /devices/virtual/block/loop0 (block)
UDEV [3528.124410] change /devices/virtual/block/loop0 (block)
UDEV [3528.144352] change /devices/virtual/block/loop0 (block)
KERNEL[3528.148718] change /devices/virtual/block/nbd0 (block)
KERNEL[3528.151216] change /devices/virtual/block/nbd0 (block)
KERNEL[3528.151302] add /devices/virtual/block/nbd0/nbd0p1 (block)
UDEV [3528.163222] change /devices/virtual/block/nbd0 (block)
UDEV [3528.167442] change /devices/virtual/block/nbd0 (block)
UDEV [3528.173677] add /devices/virtual/block/nbd0/nbd0p1 (block)
KERNEL[3528.267889] change /devices/virtual/block/nbd0 (block)
KERNEL[3528.269940] remove /devices/virtual/block/nbd0/nbd0p1 (block)
UDEV [3528.275148] change /devices/virtual/block/nbd0 (block)
UDEV [3528.276144] remove /devices/virtual/block/nbd0/nbd0p1 (block)

Stefan Bader (smb) wrote :

Somehow looks like the creation of the nbd device does not reliably trigger the partition scanning in kernel. The following work-around would at least here get past the problem:

--- /usr/bin/mount-image-callback 2016-09-01 22:07:51.000000000 +0200
+++ mount-image-callback 2016-09-28 11:46:29.404822620 +0200
@@ -321,6 +321,7 @@
     debug 1 "unpartitioned disk."
     debug 1 "partitioned disk."
+ blockdev --rereadpt $nbd

Stefan Bader (smb) on 2016-09-28
Changed in linux (Ubuntu):
assignee: nobody → Stefan Bader (smb)
importance: Undecided → Medium
Scott Moser (smoser) on 2016-09-28
description: updated
Tim Gardner (timg-tpi) on 2016-09-29
Changed in linux (Ubuntu Yakkety):
status: Confirmed → Fix Committed
Stefan Bader (smb) wrote :

Looking further into it, this is somehow caused by changes to the nbd module. Before 4.8 it was actually not creating any uevents on its own. Still there were events to be seen. With 4.8 they code changed to generate a change event, but also will delay the capacity change until the moment the socket is connected. And this seems to prevent whatever is otherwise producing the change event and triggering a partition scan. The nbd driver itself does not do that (like for example the loop driver does). So it could be that this had ever worked only because of the mysterious helper.

It seems that reverting a bit back to change the capacity as soon as it is set by ioctl (there are three ioctls which do this, so the values likely get touched multiple times, too) will again get the partition detected reliably. Not sure this can be guaranteed always...

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 4.8.0-19.21

linux (4.8.0-19.21) yakkety; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1629057

  * 4.8.0 kernels do not complete boot process on VM (LP: #1627198)

  * mount-image-callback cannot mount partitioned disk image (LP: #1628336)
    - SAUCE: nbd: Only delay uevent until connected

  * Support snaps inside of lxd containers (LP: #1611078)
    - apparmor: add interface to be able to grab loaded policy
    - securityfs: update interface to allow inode_ops, and setup from vfs fns
    - apparmor: refactor aa_prepare_ns into prepare_ns and create_ns routines
    - apparmor: add __aa_find_ns fn
    - apparmor: add mkdir/rmdir interface to manage policy namespaces
    - apparmor: fix oops in pivot_root mediation
    - apparmor: fix warning that fn build_pivotroot discards const
    - apparmor: add interface to advertise status of current task stacking
    - apparmor: update policy permissions to consider ns being viewed/managed
    - apparmor: add per ns policy management interface
    - apparmor: bump domain stacking version to 1.2

  * linux-image-extra-4.8.0-17-generic does not provide many sound card modules
    (LP: #1628523)
    - [Config] CONFIG_ZONE_DMA=y for generic

  * Yakkety - disable ARCH_ZX (LP: #1628503)
    - [Config] armhf: disable ARCH_ZX

  * Enable switchdev config parameter for Yakkety (LP: #1628241)
    - [Config] CONFIG_NET_SWITCHDEV=y for amd64/arm64

  * Ubuntu 16.10 kernel v4.8: Installation failing on Habanero with Shiner card
    (LP: #1628009)
    - firmware: Update bnx2x to

  * vNIC driver missing in 4.8 kernel package (LP: #1628187)
    - [Config] Enable CONFIG_IBMVNIC=m

  * Yakkety - armhf: MFD_TPS65217 and REGULATOR_TPS65217 are boot essential
    (LP: #1628112)
    - [Config] armhf: MFD_TPS65217=y && REGULATOR_TPS65217=y

  * Miscellaneous Ubuntu changes
    - Rebase to v4.8-rc8
    - [Config] skip Ubuntu-4.8.0-18.20
    - [Config] missing modules in armhf/s390x

  * Miscellaneous Ubuntu changes
    - rebase to v4.8-rc8

 -- Leann Ogasawara <email address hidden> Sun, 25 Sep 2016 12:13:35 -0700

Changed in linux (Ubuntu Yakkety):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers