Comment 2 for bug 1713099
Revision history for this message
| Dmitrii Shcherbakov (dmitriis) wrote Re: ceph + pike - ceph-disk prepare fails when directory specified rather than block device | #2 |
© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
0e1f616
(Get the code!)
TL;DR: it is possible to prepare and activate an osd with bluestore backend on top of regular files by using a directory path during ceph-disk invocations for testing purposes. BlueStore handles that properly which I have tested on my virtual environment.
In this case one of 2 open(2) or posix_fadvise(2) calls failed in KernelDevice:
http://
http://
https:/
Given that I was not able to reproduce it on my environment with just using the same ceph-disk prepare and ceph-disk activate calls I am assuming that something is wrong with the directory that was specified in the charm or the file system it resides on.
open(2) EINVAL returns if:
EINVAL The filesystem does not support the O_DIRECT flag. See NOTES
for more information.
EINVAL Invalid value in flags.
EINVAL O_TMPFILE was specified in flags, but neither O_WRONLY nor
posix_fadvise(2)
EINVAL An invalid value was specified for advice.
2 things we need to do:
1. Modify osdize_dir to pass filestore as a default as we've used the same approach for the block device code path
2. Get an environment where issue is triggered and get contents of /proc/mounts, an output of `stat /srv/osd` and an output of /srv/osd/block after the issue is triggered.
The rest of the analysis is below.
---
bluestore is the new default objectstore and this is the commit for ceph-disk that does the change:
https:/
Now, as for our code:
unit-ceph-3: 14:56:53 INFO unit.ceph/
Even as of jewel data_dir doesn't really do anything in ceph-disk:
➜ ceph_disk git:(jewel) grep -RiP 'data.*?dir'
main.py: '--data-dir',
main.py: help='verify that DATA is a dir',
main.py: help='path to OSD data (a disk block device or directory)',
main.py: LOG.debug('Data dir %s already exists', path)
main.py: LOG.debug(
main.py: raise Error('data path for directory does not exist',
main.py: LOG.debug('%s osd.%s data dir is ready at %s', cluster, osd_id, path)
It used to mean something in hammer:
➜ src git:(hammer) grep -iP 'data.*?dir' ceph-disk
if args.data_dir:
if args.data_dir:
LOG.debug('%s osd.%s data dir is ready at %s', cluster, osd_id, path)
help='path to OSD data (a disk block device or directory)',
Either way, ceph-disk in Luminous uses the following class hierarchy:
PrepareSpace <- PrepareData <- PrepareFilestor
PrepareSpace <- PrepareData <- PrepareBluestor
Some code relevant to how directory detection is translated to creating files that will actually store data regardless of the object store type (filestore or bluestore):
PrepareSpace
https:/
has 'def prepare' which acts differently upon file type (regular or block special):
https:/
If stat.S_ISDIR -> self.type = self.FILE
https:/
prepare -> prepare_file
https:/
https:/
Type of a file is checked via regular stat(2):
https:/
https:/
activate_dir is also performed based upon stat(2):
https:/
https:/
Now, from the BlueStore's perspective itself it is also : https:/
https:/
https:/
https:/
----
And a practical test:
http://
----
Looking more at the error messages:
unit-ceph-3: 14:57:01 INFO unit.ceph/
unit-ceph-3: 14:57:01 INFO unit.ceph/
unit-ceph-3: 14:57:01 INFO unit.ceph/
unit-ceph-3: 14:57:01 INFO unit.ceph/
This error is important: "open open got: (22) Invalid argument"
That's what was called:
bdev(0xaaab03430780 /srv/osd/block)
https:/
https:/
virtual int open(const std::string& path) = 0;
pure virtual function overriden in "class KernelDevice : public BlockDevice"
https:/
dout prefix of our error "bdev(" << this << " " << path << ")" - address of this KernelDevice object and an argument:
https:/
There are 3 places which can output such an error:
fd_direct = ::open(
https:/
fd_buffered = ::open(
https:/
r = posix_fadvise(
https:/
One of those failed with EINVAL.