generic driver: Creating regular file /dev/vdc

Bug #1547000 reported by Thomas Bechtold
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Undecided
Thomas Bechtold

Bug Description

When using the generic driver (with liberty), there's a race when attaching a volume and then use the attached volume inside the VM to create a filesystem. When mkfs is called before the kernel from the service VM creates the device node (i.e. /dev/vdc) then mkfs creates a new file with size 0. The error then is:

2016-02-16 18:03:08.182 14004 ERROR manila.share.manager [req-6c441864-65d7-4e11-aaa9-65b6c9529801 ] Share instance ebae33cc-3e1b-494e-a6a1-ae54ec830c9a failed on creation.
2016-02-16 18:03:08.183 14004 WARNING manila.share.manager [req-6c441864-65d7-4e11-aaa9-65b6c9529801 ] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher [req-6c441864-65d7-4e11-aaa9-65b6c9529801 ] Exception during message handling: Unexpected error while running command.
Command: None
Exit code: -
Stdout: u'Unexpected error while running command.\nCommand: sudo mkfs.ext3 /dev/vdc\nExit code: 1\nStdout: u\'Creating regular file /dev/vdc\\n\'\nStderr: u"mke2fs 1.42.11 (09-Jul-2014)\\nmkfs.ext3: Device size reported to be zero. Invalid partition specified, or\\n\\tpartition table wasn\'t reread after running fdisk, due to\\n\\ta modified partition being busy and in use. You may need to reboot\\n\\tto re-read your partition table.\\n\\n"'
Stderr: None
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 110, in wrapped
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher return f(self, *args, **kwargs)
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 767, in create_share_instance
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher {'status': constants.STATUS_ERROR}
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/manila/share/manager.py", line 738, in create_share_instance
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher context, share_instance, share_server=share_server)
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/manila/share/drivers/generic.py", line 123, in wrap
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kwargs)
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/manila/share/drivers/generic.py", line 218, in create_share
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher self._format_device(server_details, volume)
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/manila/share/drivers/generic.py", line 229, in _format_device
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher self._ssh_exec(server_details, command)
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/manila/share/drivers/generic.py", line 168, in _ssh_exec
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher return processutils.ssh_execute(ssh, ' '.join(command))
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 393, in ssh_execute
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher cmd=sanitized_cmd)
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher Command: sudo mkfs.ext3 /dev/vdc
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher Exit code: 1
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher Stdout: u'Creating regular file /dev/vdc\n'
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher Stderr: u"mke2fs 1.42.11 (09-Jul-2014)\nmkfs.ext3: Device size reported to be zero. Invalid partition specified, or\n\tpartition table wasn't reread after running fdisk, due to\n\ta modified partition being busy and in use. You may need to reboot\n\tto re-read your partition table.\n\n"
2016-02-16 18:03:08.222 14004 ERROR oslo_messaging.rpc.dispatcher

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.openstack.org/281849

Changed in manila:
assignee: nobody → Thomas Bechtold (toabctl)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.openstack.org/281849
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=b8e3f981a722e0d815e9acfddd88bd4938b22a42
Submitter: Jenkins
Branch: master

commit b8e3f981a722e0d815e9acfddd88bd4938b22a42
Author: Thomas Bechtold <email address hidden>
Date: Thu Feb 18 14:26:45 2016 +0100

    Check for device node availability before mkfs

    When using the generic driver, a volume is attached to the service
    VM and then a mkfs is done to create a FS on that volume inside of
    the VM. But it can be the case that it takes a bit of time before
    the device node pops up inside of the VM. So wait some seconds
    before doing a mkfs on a non-existing device node.

    Change-Id: I42228d9a8ad836a95be5c59e43bd6762677dd85f
    Closes-bug: #1547000

Changed in manila:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.