mkfs.xfs: cannot open /dev/sdb: Device or resource busy

Bug #1513204 reported by Ryan Beisner on 2015-11-04
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack swift-storage charm
Undecided
Unassigned
swift-storage (Juju Charms Collection)
Undecided
Unassigned

Bug Description

The mkfs.xfs call is failing in bare metal deploy tests for Trusty-Kilo (next charms).

http://paste.ubuntu.com/13105318/

2015-11-04 20:06:24 INFO install mkfs.xfs: cannot open /dev/sdb: Device or resource busy
2015-11-04 20:06:24 INFO install Traceback (most recent call last):
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/install.real", line 175, in <module>
2015-11-04 20:06:24 INFO install main()
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/install.real", line 167, in main
2015-11-04 20:06:24 INFO install hooks.execute(sys.argv)
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/charmhelpers/core/hookenv.py", line 704, in execute
2015-11-04 20:06:24 INFO install self._hooks[hook_name]()
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/install.real", line 70, in install
2015-11-04 20:06:24 INFO install setup_storage()
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/lib/swift_storage_utils.py", line 252, in setup_storage
2015-11-04 20:06:24 INFO install mkfs_xfs(dev)
2015-11-04 20:06:24 INFO install File "/var/lib/juju/agents/unit-swift-storage-z2-0/charm/hooks/lib/swift_storage_utils.py", line 244, in mkfs_xfs
2015-11-04 20:06:24 INFO install check_call(cmd)
2015-11-04 20:06:24 INFO install File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
2015-11-04 20:06:24 INFO install raise CalledProcessError(retcode, cmd)
2015-11-04 20:06:24 INFO install subprocess.CalledProcessError: Command '['mkfs.xfs', '-f', '-i', 'size=1024', u'/dev/sdb']' returned non-zero exit status 1
2015-11-04 20:06:24 INFO juju.worker.uniter.context context.go:579 handling reboot
2015-11-04 20:06:24 ERROR juju.worker.uniter.operation runhook.go:107 hook "install" failed: exit status 1

Related branches

Ryan Beisner (1chb1n) wrote :

After waiting a while, I was able to manually run the same mkfs command on the unit:

http://paste.ubuntu.com/13105459/

ubuntu@international-steam:~$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
└─sda1 8:1 0 465.8G 0 part /
sdb 8:16 0 465.8G 0 disk
sr0 11:0 1 1024M 0 rom

...

ubuntu@international-steam:~$ sudo parted -l
Model: ATA ST9500620NS (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
 1 1049kB 500GB 500GB primary ext4 boot

Error: /dev/sdb: unrecognised disk label

...

ubuntu@international-steam:~$ sudo mkfs.xfs -f -i size=1024 /dev/sdb
meta-data=/dev/sdb isize=1024 agcount=4, agsize=30524162 blks
         = sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=122096646, imaxpct=25
         = sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=59617, version=2
         = sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
ubuntu@international-steam:~$ echo $?
0

...

ubuntu@international-steam:~$ sudo parted -l
Model: ATA ST9500620NS (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
 1 1049kB 500GB 500GB primary ext4 boot

Model: ATA ST9500620NS (scsi)
Disk /dev/sdb: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: loop

Number Start End Size File system Flags
 1 0.00B 500GB 500GB xfs

James Page (james-page) wrote :

Possibly the addition of a udevadm settle call may avoid this race.

Ryan Beisner (1chb1n) wrote :

@jamespage

This is odd. I added udevadm settle like so, so that it is called before the storage operations:
lp:~openstack-charmers/charms/trusty/swift-storage/next

The udevadm command itself errored.

When I went back into the unit, I was able to do the udevadm commands and such without error.

I've not dug deeper than that. I'll hold on this, pending your input.

FWIW, when I arrived at the unit, a previous deployment's ceph gpt was still present on /dev/sdb:

root@decisive-punishment:/var/log# sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 465.8G 0 disk
└─sda1 8:1 0 465.8G 0 part /
sdb 8:16 0 465.8G 0 disk
├─sdb1 8:17 0 464.8G 0 part
└─sdb2 8:18 0 1023M 0 part
sr0 11:0 1 1024M 0 rom

root@decisive-punishment:/var/log# sudo parted -l
Model: ATA ST9500620NS (scsi)
Disk /dev/sda: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
 1 1049kB 500GB 500GB primary ext4 boot

Model: ATA ST9500620NS (scsi)
Disk /dev/sdb: 500GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 2 1049kB 1074MB 1073MB ceph journal
 1 1075MB 500GB 499GB xfs ceph data

James Page (james-page) on 2017-02-23
Changed in swift-storage (Juju Charms Collection):
status: New → Invalid
James Page (james-page) wrote :

This bug appears to relate to a transient kernel bug that we saw around this time; We don't seen this now with 16.04 and later testing so marking this bug as invalid.

Changed in charm-swift-storage:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers