Race condition: rbd device not yet available when mkfs is called

Bug #1210309 reported by Andreas Hasenack
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
mysql (Juju Charms Collection)
Fix Released
Undecided
Unassigned

Bug Description

Same bug as https://bugs.launchpad.net/charms/+source/rabbitmq-server/+bug/1210267

When establishing a relation with ceph, there is a race condition in which the rbd device is not yet available but the charm attempts to format it:
2013-08-08 21:42:19 INFO juju juju-log.go:64 mysql/0 ceph:7: ceph: Creating new pool mysql.
2013-08-08 21:42:19 INFO juju context.go:235 worker/uniter: HOOK successfully created pool mysql
2013-08-08 21:42:20 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["--log-level" "INFO" "ceph: Creating RBD image (mysql1)."]
2013-08-08 21:42:20 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "mysql/0:ceph-relation-changed:3013356756796808072"; dir "/var/lib/juju/agents/unit-mysql-0/charm"
2013-08-08 21:42:20 INFO juju juju-log.go:64 mysql/0 ceph:7: ceph: Creating RBD image (mysql1).
2013-08-08 21:42:20 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["--log-level" "INFO" "ceph: Mapping RBD Image as a Block Device."]
2013-08-08 21:42:20 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "mysql/0:ceph-relation-changed:3013356756796808072"; dir "/var/lib/juju/agents/unit-mysql-0/charm"
2013-08-08 21:42:20 INFO juju juju-log.go:64 mysql/0 ceph:7: ceph: Mapping RBD Image as a Block Device.
2013-08-08 21:42:20 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["--log-level" "INFO" "ceph: Formatting block device /dev/rbd/mysql/mysql1 as filesystem ext4."]
2013-08-08 21:42:20 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "mysql/0:ceph-relation-changed:3013356756796808072"; dir "/var/lib/juju/agents/unit-mysql-0/charm"
2013-08-08 21:42:20 INFO juju juju-log.go:64 mysql/0 ceph:7: ceph: Formatting block device /dev/rbd/mysql/mysql1 as filesystem ext4.
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK mke2fs 1.42 (29-Nov-2011)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK Could not stat /dev/rbd/mysql/mysql1 --- No such file or directory
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK The device apparently does not exist; did you specify it correctly?
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK Traceback (most recent call last):
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/ceph-relation-changed", line 146, in <module>
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK utils.do_hooks(hooks)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/utils.py", line 28, in do_hooks
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK hook_func()
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/ceph-relation-changed", line 117, in ceph_changed
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK system_services=['mysql'])
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/ceph_utils.py", line 244, in ensure_ceph_storage
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK make_filesystem(blk_device, fstype)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/ceph_utils.py", line 160, in make_filesystem
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK execute(cmd)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/ceph_utils.py", line 28, in execute
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK subprocess.check_call(cmd)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK raise CalledProcessError(retcode, cmd)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK subprocess.CalledProcessError: Command '['mkfs', '-t', 'ext4', u'/dev/rbd/mysql/mysql1']' returned non-zero exit status 1
2013-08-08 21:42:20 ERROR juju uniter.go:352 worker/uniter: hook failed: exit status 1

Tags: landscape

Related branches

Changed in mysql (Juju Charms Collection):
status: New → Fix Released
tags: added: landscape
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.