Race condition: rbd device not yet available when mkfs is called

[Andreas Hasenack]

Same bug as https://bugs.launchpad.net/charms/+source/rabbitmq-server/+bug/1210267

When establishing a relation with ceph, there is a race condition in which the rbd device is not yet available but the charm attempts to format it:
2013-08-08 21:42:19 INFO juju juju-log.go:64 mysql/0 ceph:7: ceph: Creating new pool mysql.
2013-08-08 21:42:19 INFO juju context.go:235 worker/uniter: HOOK successfully created pool mysql
2013-08-08 21:42:20 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["--log-level" "INFO" "ceph: Creating RBD image (mysql1)."]
2013-08-08 21:42:20 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "mysql/0:ceph-relation-changed:3013356756796808072"; dir "/var/lib/juju/agents/unit-mysql-0/charm"
2013-08-08 21:42:20 INFO juju juju-log.go:64 mysql/0 ceph:7: ceph: Creating RBD image (mysql1).
2013-08-08 21:42:20 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["--log-level" "INFO" "ceph: Mapping RBD Image as a Block Device."]
2013-08-08 21:42:20 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "mysql/0:ceph-relation-changed:3013356756796808072"; dir "/var/lib/juju/agents/unit-mysql-0/charm"
2013-08-08 21:42:20 INFO juju juju-log.go:64 mysql/0 ceph:7: ceph: Mapping RBD Image as a Block Device.
2013-08-08 21:42:20 INFO juju server.go:105 worker/uniter/jujuc: running hook tool "juju-log" ["--log-level" "INFO" "ceph: Formatting block device /dev/rbd/mysql/mysql1 as filesystem ext4."]
2013-08-08 21:42:20 DEBUG juju server.go:106 worker/uniter/jujuc: hook context id "mysql/0:ceph-relation-changed:3013356756796808072"; dir "/var/lib/juju/agents/unit-mysql-0/charm"
2013-08-08 21:42:20 INFO juju juju-log.go:64 mysql/0 ceph:7: ceph: Formatting block device /dev/rbd/mysql/mysql1 as filesystem ext4.
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK mke2fs 1.42 (29-Nov-2011)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK Could not stat /dev/rbd/mysql/mysql1 --- No such file or directory
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK The device apparently does not exist; did you specify it correctly?
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK Traceback (most recent call last):
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/ceph-relation-changed", line 146, in <module>
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK utils.do_hooks(hooks)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/utils.py", line 28, in do_hooks
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK hook_func()
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/ceph-relation-changed", line 117, in ceph_changed
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK system_services=['mysql'])
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/ceph_utils.py", line 244, in ensure_ceph_storage
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK make_filesystem(blk_device, fstype)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/ceph_utils.py", line 160, in make_filesystem
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK execute(cmd)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/var/lib/juju/agents/unit-mysql-0/charm/hooks/lib/ceph_utils.py", line 28, in execute
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK subprocess.check_call(cmd)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK raise CalledProcessError(retcode, cmd)
2013-08-08 21:42:20 INFO juju context.go:235 worker/uniter: HOOK subprocess.CalledProcessError: Command '['mkfs', '-t', 'ext4', u'/dev/rbd/mysql/mysql1']' returned non-zero exit status 1
2013-08-08 21:42:20 ERROR juju uniter.go:352 worker/uniter: hook failed: exit status 1