[disco] [eoan] After unmount, cannot open /dev/vdb: Device or resource busy
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Cinder Charm |
Triaged
|
Undecided
|
Unassigned | ||
OpenStack Nova Compute Charm |
Triaged
|
Undecided
|
Unassigned | ||
linux (Ubuntu) |
Confirmed
|
Undecided
|
Unassigned |
Bug Description
Disco and Eoan device is busy after unmounting an ephemeral disk, cannot format the device until rebooting.
This is blocking all of OpenStack Charms which interact with block devices (Nova Compute, Ceph, Swift, Cinder), on the Disco and Eoan series. As we are nearing LTS-1, this will become urgent pretty quickly.
Reproducer, on an OpenStack cloud:
juju deploy cs:ubuntu ubuntu-bionic --series bionic
juju deploy cs:ubuntu ubuntu-disco --series disco
juju deploy cs:ubuntu ubuntu-eoan --series eoan
#### Succeeds on Bionic:
ubuntu@
Linux juju-8d01b7-foo-14 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@
Codename: bionic
ubuntu@
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk /mnt
ubuntu@
Filesystem Size Used Avail Use% Mounted on
udev 985M 0 985M 0% /dev
tmpfs 200M 712K 199M 1% /run
/dev/vda1 20G 1.7G 18G 9% /
tmpfs 997M 0 997M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 997M 0 997M 0% /sys/fs/cgroup
/dev/vda15 105M 3.6M 101M 4% /boot/efi
/dev/vdb 40G 49M 38G 1% /mnt
tmpfs 100K 0 100K 0% /var/lib/
tmpfs 100K 0 100K 0% /var/lib/lxd/devlxd
tmpfs 200M 0 200M 0% /run/user/1000
ubuntu@
ubuntu@
meta-data=/dev/vdb isize=1024 agcount=4, agsize=2621440 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0
data = bsize=4096 blocks=10485760, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=5120, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
#### Fails on Disco:
ubuntu@
Linux juju-8d01b7-foo-12 5.0.0-27-generic #28-Ubuntu SMP Tue Aug 20 19:53:07 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@
Codename: disco
ubuntu@
ubuntu@
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 88.7M 1 loop /snap/core/7396
loop1 7:1 0 54.5M 1 loop /snap/lxd/11727
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk
ubuntu@
mkfs.xfs: cannot open /dev/vdb: Device or resource busy
#### Fails on Eoan:
ubuntu@
ubuntu@
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 88.7M 1 loop /snap/core/7396
loop1 7:1 0 54.5M 1 loop /snap/lxd/11727
vda 252:0 0 20G 0 disk
├─vda1 252:1 0 19.9G 0 part /
├─vda14 252:14 0 4M 0 part
└─vda15 252:15 0 106M 0 part /boot/efi
vdb 252:16 0 40G 0 disk
ubuntu@
mkfs.xfs: cannot open /dev/vdb: Device or resource busy
..
Original bug description:
On disco-stein, hook failed: "config-changed" with mkfs.xfs: cannot open /dev/vdb: Device or resource busy
2019-09-03 21:33:27 DEBUG config-changed Unpacking apparmor-utils (2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up python3-libapparmor (2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up python3-apparmor (2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Setting up apparmor-utils (2.13.2-9ubuntu6.1) ...
2019-09-03 21:33:27 DEBUG config-changed Processing triggers for man-db (2.8.5-2) ...
2019-09-03 21:33:30 INFO juju-log Setting up the apparmor profile for usr.bin.
2019-09-03 21:33:30 DEBUG config-changed Setting /etc/apparmor.
2019-09-03 21:33:30 DEBUG config-changed mkfs.xfs: cannot open /dev/vdb: Device or resource busy
2019-09-03 21:33:30 DEBUG config-changed Traceback (most recent call last):
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/
2019-09-03 21:33:30 DEBUG config-changed main()
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/
2019-09-03 21:33:30 DEBUG config-changed hooks.execute(
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/
2019-09-03 21:33:30 DEBUG config-changed self._hooks[
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/
2019-09-03 21:33:30 DEBUG config-changed stopstart, restart_functions)
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/
2019-09-03 21:33:30 DEBUG config-changed r = lambda_f()
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/
2019-09-03 21:33:30 DEBUG config-changed (lambda: f(*args, **kwargs)), __restart_
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/
2019-09-03 21:33:30 DEBUG config-changed return f(*args, **kwargs)
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/
2019-09-03 21:33:30 DEBUG config-changed configure_
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/
2019-09-03 21:33:30 DEBUG config-changed mkfs_xfs(dev, force=True)
2019-09-03 21:33:30 DEBUG config-changed File "/var/lib/
2019-09-03 21:33:30 DEBUG config-changed check_call(cmd)
2019-09-03 21:33:30 DEBUG config-changed File "/usr/lib/
2019-09-03 21:33:30 DEBUG config-changed raise CalledProcessEr
2019-09-03 21:33:30 DEBUG config-changed subprocess.
2019-09-03 21:33:30 ERROR juju.worker.
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Sep 5 20:47 seq
crw-rw---- 1 root audio 116, 33 Sep 5 20:47 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.10-0ubuntu27.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 19.04
Ec2AMI: ami-0000078c
Ec2AMIManifest: FIXME
Ec2Availability
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=screen-
PATH=(custom, no user)
XDG_RUNTIME_
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: disco ec2-images
Uname: Linux 5.0.0-27-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.
dmi.modalias: dmi:bvnSeaBIOS:
dmi.product.family: Virtual Machine
dmi.product.name: OpenStack Nova
dmi.product.
dmi.sys.vendor: OpenStack Foundation
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Sep 5 20:48 seq
crw-rw---- 1 root audio 116, 33 Sep 5 20:48 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.11-0ubuntu7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 19.10
Ec2AMI: ami-0000078d
Ec2AMIManifest: FIXME
Ec2Availability
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU USB Tablet
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=screen-
PATH=(custom, no user)
XDG_RUNTIME_
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: eoan ec2-images
Uname: Linux 5.2.0-15-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.
dmi.modalias: dmi:bvnSeaBIOS:
dmi.product.family: Virtual Machine
dmi.product.name: OpenStack Nova
dmi.product.
dmi.sys.vendor: OpenStack Foundation
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Sep 5 20:48 seq
crw-rw---- 1 root audio 116, 33 Sep 5 20:48 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.7
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 18.04
Ec2AMI: ami-00000789
Ec2AMIManifest: FIXME
Ec2Availability
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
Lsusb:
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=screen-
PATH=(custom, no user)
XDG_RUNTIME_
LANG=C.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
Tags: bionic ec2-images
Uname: Linux 4.15.0-60-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: adm audio cdrom dialout dip floppy netdev plugdev sudo video
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.
dmi.modalias: dmi:bvnSeaBIOS:
dmi.product.family: Virtual Machine
dmi.product.name: OpenStack Nova
dmi.product.
dmi.sys.vendor: OpenStack Foundation
summary: |
- disco-stein: hook failed: "config-changed" with mkfs.xfs: cannot open - /dev/vdb: Device or resource busy + [disco] mkfs.xfs: cannot open /dev/vdb: Device or resource busy |
summary: |
- [disco] mkfs.xfs: cannot open /dev/vdb: Device or resource busy + [disco] [eoan] After unmount, mkfs.xfs: cannot open /dev/vdb: Device or + resource busy |
description: | updated |
summary: |
- [disco] [eoan] After unmount, mkfs.xfs: cannot open /dev/vdb: Device or - resource busy + [disco] [eoan] After unmount, cannot open /dev/vdb: Device or resource + busy |
Changed in charm-cinder: | |
status: | New → Triaged |
Changed in charm-nova-compute: | |
status: | New → Triaged |
Affects similarly charm-cinder. https:/ /openstack- ci-reports. ubuntu. com/artifacts/ test_charm_ pipeline_ func_full/ openstack/ charm-cinder/ 678676/ 3/3803/ index.html
root@juju- ccba06- zaza-ea7ecb946e bc-0:/var/ lib/juju/ agents/ unit-cinder- 0/charm# ./hooks/ config- changed config- changed" , line 622, in <module> execute( sys.argv) juju/agents/ unit-cinder- 0/charm/ charmhelpers/ core/hookenv. py", line 914, in execute _hooks[ hook_name] () juju/agents/ unit-cinder- 0/charm/ charmhelpers/ contrib/ openstack/ utils.py" , line 1560, in wrapped_f juju/agents/ unit-cinder- 0/charm/ charmhelpers/ core/host. py", line 741, in restart_ on_change_ helper juju/agents/ unit-cinder- 0/charm/ charmhelpers/ contrib/ openstack/ utils.py" , line 1559, in <lambda> map_cache[ 'cache' ], juju/agents/ unit-cinder- 0/charm/ charmhelpers/ contrib/ hardening/ harden. py", line 93, in _harden_inner2 config- changed" , line 197, in config_changed 'remove- missing- force'] ) juju/agents/ unit-cinder- 0/charm/ hooks/cinder_ utils.py" , line 543, in configure_ lvm_storage volume( device) juju/agents/ unit-cinder- 0/charm/ hooks/cinder_ utils.py" , line 594, in prepare_volume lvm_physical_ volume( device) juju/agents/ unit-cinder- 0/charm/ charmhelpers/ contrib/ storage/ linux/lvm. py", line 92, in create_ lvm_physical_ volume call([' pvcreate' , block_device]) python3. 7/subprocess. py", line 347, in check_call ror(retcode, cmd) CalledProcessEr ror: Command '['pvcreate', '/dev/vdb']' returned non-zero exit status 5.
Failed to find physical volume "/dev/vdb".
Failed to find physical volume "/dev/vdb".
Creating new GPT entries.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00690306 s, 152 MB/s
100+0 records in
100+0 records out
51200 bytes (51 kB, 50 KiB) copied, 0.00156154 s, 32.8 MB/s
Can't open /dev/vdb exclusively. Mounted filesystem?
Traceback (most recent call last):
File "./hooks/
hooks.
File "/var/lib/
self.
File "/var/lib/
stopstart, restart_functions)
File "/var/lib/
r = lambda_f()
File "/var/lib/
(lambda: f(*args, **kwargs)), __restart_
File "/var/lib/
return f(*args, **kwargs)
File "./hooks/
conf[
File "/var/lib/
prepare_
File "/var/lib/
create_
File "/var/lib/
check_
File "/usr/lib/
raise CalledProcessEr
subprocess.