Error: Device is mounted

[Jason Hobbs]

It looks like the ceph-osd charm is trying to initialize my storage twice:

2018-01-29 22:35:21 INFO juju-log mon:43: osdize cmd: ['ceph-disk', 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore', '/dev/sdb']

2018-01-29 22:36:03 INFO juju-log mon:43: osdize cmd: ['ceph-disk', 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore', '/dev/sdb']

The second time in fails with an error:
2018-01-29 22:36:03 DEBUG mon-relation-changed ceph-disk: Error: Device is mounted: /dev/sdb1

ceph-osd/5 log: http://paste.ubuntu.com/26486128/

bundle: http://paste.ubuntu.com/26486149/

[Christian Reis (kiko)]

2018-01-29 22:36:04 ERROR juju-log mon:43: Unable to initialize device: /dev/sdb
2018-01-29 22:36:04 DEBUG mon-relation-changed Traceback (most recent call last):
2018-01-29 22:36:04 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-5/charm/hooks/mon-relation-changed", line 559, in <module>
2018-01-29 22:36:04 DEBUG mon-relation-changed hooks.execute(sys.argv)
2018-01-29 22:36:04 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-5/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-01-29 22:36:04 DEBUG mon-relation-changed self._hooks[hook_name]()
2018-01-29 22:36:04 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-5/charm/hooks/mon-relation-changed", line 486, in mon_relation
2018-01-29 22:36:04 DEBUG mon-relation-changed prepare_disks_and_activate()
2018-01-29 22:36:04 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-5/charm/hooks/mon-relation-changed", line 389, in prepare_disks_and_activate
2018-01-29 22:36:04 DEBUG mon-relation-changed config('bluestore'))
2018-01-29 22:36:04 DEBUG mon-relation-changed File "lib/ceph/utils.py", line 1436, in osdize
2018-01-29 22:36:04 DEBUG mon-relation-changed bluestore)
2018-01-29 22:36:04 DEBUG mon-relation-changed File "lib/ceph/utils.py", line 1504, in osdize_dev
2018-01-29 22:36:04 DEBUG mon-relation-changed subprocess.check_call(cmd)
2018-01-29 22:36:04 DEBUG mon-relation-changed File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
2018-01-29 22:36:04 DEBUG mon-relation-changed raise CalledProcessError(retcode, cmd)
2018-01-29 22:36:04 DEBUG mon-relation-changed subprocess.CalledProcessError: Command '['ceph-disk', 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore', '/dev/sdb']' returned non-zero exit status 1
2018-01-29 22:36:04 ERROR juju.worker.uniter.operation runhook.go:114 hook "mon-relation-changed" failed: exit status 1

[Nobuto Murata (nobuto)]

Somehow, "is_device_mounted" check didn't work before running "ceph-disk prepare"?

def osdize_dev(dev, osd_format, osd_journal, reformat_osd=False,
               ignore_errors=False, encrypt=False, bluestore=False):
    if not os.path.exists(dev):
        log('Path {} does not exist - bailing'.format(dev))
        return

    if not is_block_device(dev):
        log('Path {} is not a block device - bailing'.format(dev))
        return

    if is_osd_disk(dev) and not reformat_osd:
        log('Looks like {} is already an'
            ' OSD data or journal, skipping.'.format(dev))
        return

    if is_device_mounted(dev):
        log('Looks like {} is in use, skipping.'.format(dev))
        return

    status_set('maintenance', 'Initializing device {}'.format(dev))
    cmd = ['ceph-disk', 'prepare']

[Christian Reis (kiko)]

Looks like the easy way to solve this is to do check for is_device_mounted at the end of osdize_dev() and raise an error if it's not.

[Christian Reis (kiko)]

We believe the race here is the sysfs nodes not being set up in time for lsblk to return the right data, which is_device_mounted() uses to confirm the mount is present.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/544972
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-osd/commit/?id=eeacba1614859c269ae8d7fa652443edf9eda211
Submitter: Zuul
Branch: master

commit eeacba1614859c269ae8d7fa652443edf9eda211
Author: Chris MacNaughton <email address hidden>
Date: Thu Feb 15 15:03:07 2018 +0100

    Sync in charms.ceph change for udev settle

    Change-Id: Ideb8dbe8e6e43966baa83084fa0ea7eac2e2597c
    Closes-Bug: #1746118

[Jason Hobbs (jason-hobbs)]

We hit this bug again yesterday running against the tip version of the charms.

[Jason Hobbs (jason-hobbs)]

Marked back to 'new' as the commit above didn't seem to fix this issue.

[Chris Gregan (cgregan)]

Escalated to Field High to increase visibility

[Christian Reis (kiko)]

Dupe of bug 1751127?

[Jason Hobbs (jason-hobbs)]

No, Chris said they are separate issues.

On Wed, Mar 7, 2018 at 10:21 AM, Christian Reis <email address hidden> wrote:
> Dupe of bug 1751127?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1746118
>
> Title:
> Error: Device is mounted
>
> Status in OpenStack ceph-osd charm:
> New
>
> Bug description:
> It looks like the ceph-osd charm is trying to initialize my storage
> twice:
>
> 2018-01-29 22:35:21 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> 2018-01-29 22:36:03 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> The second time in fails with an error:
> 2018-01-29 22:36:03 DEBUG mon-relation-changed ceph-disk: Error: Device is mounted: /dev/sdb1
>
> ceph-osd/5 log: http://paste.ubuntu.com/26486128/
>
> bundle: http://paste.ubuntu.com/26486149/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-ceph-osd/+bug/1746118/+subscriptions

[Ashley Lai (alai)]

Our test run pointed to cs:~chris.macnaughton/ceph-osd-6 and hit a failure in ceph-osd/3. This run is a blue store.

2018-03-07 12:45:39 DEBUG mon-relation-changed Traceback (most recent call last):
2018-03-07 12:45:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-3/charm/hooks/mon-relation-changed", line 559, in <module>
2018-03-07 12:45:39 DEBUG mon-relation-changed hooks.execute(sys.argv)
2018-03-07 12:45:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-3/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-03-07 12:45:39 DEBUG mon-relation-changed self._hooks[hook_name]()
2018-03-07 12:45:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-3/charm/hooks/mon-relation-changed", line 486, in mon_relation
2018-03-07 12:45:39 DEBUG mon-relation-changed prepare_disks_and_activate()
2018-03-07 12:45:39 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-3/charm/hooks/mon-relation-changed", line 389, in prepare_disks_and_activate
2018-03-07 12:45:39 DEBUG mon-relation-changed config('bluestore'))
2018-03-07 12:45:39 DEBUG mon-relation-changed File "lib/ceph/utils.py", line 1441, in osdize
2018-03-07 12:45:39 DEBUG mon-relation-changed bluestore)
2018-03-07 12:45:39 DEBUG mon-relation-changed File "lib/ceph/utils.py", line 1509, in osdize_dev
2018-03-07 12:45:39 DEBUG mon-relation-changed subprocess.check_call(cmd)
2018-03-07 12:45:39 DEBUG mon-relation-changed File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
2018-03-07 12:45:39 DEBUG mon-relation-changed raise CalledProcessError(retcode, cmd)
2018-03-07 12:45:39 DEBUG mon-relation-changed subprocess.CalledProcessError: Command '['ceph-disk', 'prepare', '--zap-disk', '--bluestore', '/dev/sdb']' returned non-zero exit status 1

https://solutions.qa.canonical.com/#/qa/testRun/4abb4c7e-145a-4367-97a8-ca9a9e4afd78

[Chris MacNaughton (chris.macnaughton)]

cs:~chris.macnaughton/ceph-osd-7 has an update that adds a _lot_ more log noise around the lsblk

[Ashley Lai (alai)]

The charm at cs:~chris.macnaughton/ceph-osd-7 seems to fix the issue. We have not seen this issue for the past few days.

@Chris - let us know if you need a crash dump to take a look at the log. Also let us know when we should switch back to testing -next charm. Thanks!!

[Chris MacNaughton (chris.macnaughton)]

The version in my namespace shouldn't fix anything, it should only break things more noisily if it breaks. The only thing it adds is some more output around lsblk, as well as an assert to verify that the white-listed devices are mounted

[Chris MacNaughton (chris.macnaughton)]

For reference, the changes that are in my namespace will not be going into openstack-charmers-next on the charmstore...

[Ashley Lai (alai)]

It is possible that it hits other issues before getting to here. We will keep monitoring.

[Ashley Lai (alai)]

We still have not seen the issue as of today. @Chris if there are any changes in the -next charm, please rebase the patch on top of -next. That way our test will cover the new updates. Thanks !!

[Chris MacNaughton (chris.macnaughton)]

@alai there haven't been any changes to the ceph-osd charm in 29 days; the fact that you're not hitting this bug is somewhat interesting; however, nothing in the charm changes in my namespace should have any effect on that.

As a longer term solution to this issue, and other similar things that can cause the OSD to fail to be (re)created, we could wrap the ceph-disk command in some exception handling, and log failures with more information; however, I suspect that doing the above would have ended up with this bug being filed with: "No OSD devices detected with current configuration" ;-)

I'll work on updating charms.ceph to more gracefully (and verbosely) handling errors in the ceph-disk commands, at which point, re-targeting to openstack-charmers-next/ceph-osd would be advised.

[Chris MacNaughton (chris.macnaughton)]

It looks like this has been reproduced in a crashdump attached to https://bugs.launchpad.net/charm-ceph-osd/+bug/1751127/comments/10

From the juju status output in that crashdump, it looks like the ceph-osd in use is _not_ the one from my namespace, and, as such, doesn't have the additional logging added to try to diagnose this issue :-/

On the other hand, the time between successfully calling `ceph-disk` and failing the same is around 40 seconds, so not likely too fast for the kernel to see it as mounted.

[Jason Hobbs (jason-hobbs)]

Right, that crashdump came from march 2nd, before we got your updated code.

[Chris MacNaughton (chris.macnaughton)]

@jhobbs then why was it posted on 22-March?

[Jason Hobbs (jason-hobbs)]

Because you asked for a crashdump with load logs.

On Mon, Mar 26, 2018 at 2:17 AM, Chris MacNaughton
<email address hidden> wrote:
> @jhobbs then why was it posted on 22-March?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1746118
>
> Title:
> Error: Device is mounted
>
> Status in OpenStack ceph-osd charm:
> In Progress
>
> Bug description:
> It looks like the ceph-osd charm is trying to initialize my storage
> twice:
>
> 2018-01-29 22:35:21 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> 2018-01-29 22:36:03 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> The second time in fails with an error:
> 2018-01-29 22:36:03 DEBUG mon-relation-changed ceph-disk: Error: Device is mounted: /dev/sdb1
>
> ceph-osd/5 log: http://paste.ubuntu.com/26486128/
>
> bundle: http://paste.ubuntu.com/26486149/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-ceph-osd/+bug/1746118/+subscriptions

[Chris MacNaughton (chris.macnaughton)]

@jason-hobbs, @alai is this bug still occurring? If so, can we get a fresh crashdump where this bug has been reproduced along with the new load information in the crashdump?

In addition to waiting on more information for this bug, there are reviews in progress to improve logging under error conditions during disk changes in addition to improving disk management in general.

[Jason Hobbs (jason-hobbs)]

Chris, we've already attached a crashdump with load information. Does
that not have what you're looking for?

On Fri, Mar 30, 2018 at 10:05 AM, Chris MacNaughton
<email address hidden> wrote:
> @jason-hobbs, @alai is this bug still occurring? If so, can we get a
> fresh crashdump where this bug has been reproduced along with the new
> load information in the crashdump?
>
> In addition to waiting on more information for this bug, there are
> reviews in progress to improve logging under error conditions during
> disk changes in addition to improving disk management in general.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1746118
>
> Title:
> Error: Device is mounted
>
> Status in OpenStack ceph-osd charm:
> In Progress
>
> Bug description:
> It looks like the ceph-osd charm is trying to initialize my storage
> twice:
>
> 2018-01-29 22:35:21 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> 2018-01-29 22:36:03 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> The second time in fails with an error:
> 2018-01-29 22:36:03 DEBUG mon-relation-changed ceph-disk: Error: Device is mounted: /dev/sdb1
>
> ceph-osd/5 log: http://paste.ubuntu.com/26486128/
>
> bundle: http://paste.ubuntu.com/26486149/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-ceph-osd/+bug/1746118/+subscriptions

[Jason Hobbs (jason-hobbs)]

https://bugs.launchpad.net/charm-ceph-osd/+bug/1751127/comments/10 <--
load information in reproducer here

On Fri, Mar 30, 2018 at 10:12 AM, Jason Hobbs <email address hidden> wrote:
> Chris, we've already attached a crashdump with load information. Does
> that not have what you're looking for?
>
> On Fri, Mar 30, 2018 at 10:05 AM, Chris MacNaughton
> <email address hidden> wrote:
>> @jason-hobbs, @alai is this bug still occurring? If so, can we get a
>> fresh crashdump where this bug has been reproduced along with the new
>> load information in the crashdump?
>>
>> In addition to waiting on more information for this bug, there are
>> reviews in progress to improve logging under error conditions during
>> disk changes in addition to improving disk management in general.
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1746118
>>
>> Title:
>> Error: Device is mounted
>>
>> Status in OpenStack ceph-osd charm:
>> In Progress
>>
>> Bug description:
>> It looks like the ceph-osd charm is trying to initialize my storage
>> twice:
>>
>> 2018-01-29 22:35:21 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
>> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
>> '/dev/sdb']
>>
>> 2018-01-29 22:36:03 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
>> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
>> '/dev/sdb']
>>
>> The second time in fails with an error:
>> 2018-01-29 22:36:03 DEBUG mon-relation-changed ceph-disk: Error: Device is mounted: /dev/sdb1
>>
>> ceph-osd/5 log: http://paste.ubuntu.com/26486128/
>>
>> bundle: http://paste.ubuntu.com/26486149/
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/charm-ceph-osd/+bug/1746118/+subscriptions

[Chris MacNaughton (chris.macnaughton)]

The reproducer linked does _not_ have the additional logging from the charm in my namespace, meaning it is still an incomplete picture.

[Chris MacNaughton (chris.macnaughton)]

The change in https://review.openstack.org/#/c/559964 should prevent this bug from occurring any more.

[Ashley Lai (alai)]

We just hit the same issue again. ceph-osd points to cs:~chris.macnaughton/ceph-osd-7.

2018-04-14 06:06:05 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-7/charm/hooks/mon-relation-changed", line 559, in <module>
2018-04-14 06:06:05 DEBUG mon-relation-changed hooks.execute(sys.argv)
2018-04-14 06:06:05 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-7/charm/hooks/charmhelpers/core/hookenv.py", line 800, in execute
2018-04-14 06:06:05 DEBUG mon-relation-changed self._hooks[hook_name]()
2018-04-14 06:06:05 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-7/charm/hooks/mon-relation-changed", line 486, in mon_relation
2018-04-14 06:06:05 DEBUG mon-relation-changed prepare_disks_and_activate()
2018-04-14 06:06:05 DEBUG mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-7/charm/hooks/mon-relation-changed", line 389, in prepare_disks_and_activate
2018-04-14 06:06:05 DEBUG mon-relation-changed config('bluestore'))
2018-04-14 06:06:05 DEBUG mon-relation-changed File "lib/ceph/utils.py", line 1441, in osdize
2018-04-14 06:06:05 DEBUG mon-relation-changed bluestore)
2018-04-14 06:06:05 DEBUG mon-relation-changed File "lib/ceph/utils.py", line 1509, in osdize_dev
2018-04-14 06:06:05 DEBUG mon-relation-changed subprocess.check_call(cmd)
2018-04-14 06:06:05 DEBUG mon-relation-changed File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
2018-04-14 06:06:05 DEBUG mon-relation-changed raise CalledProcessError(retcode, cmd)
2018-04-14 06:06:05 DEBUG mon-relation-changed subprocess.CalledProcessError: Command '['ceph-disk', 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore', '/dev/sdb']' returned non-zero exit status 1

https://oil-jenkins.canonical.com/job/make_foundation/860/console

[Ashley Lai (alai)]

It's running Xenial Queens.

[Chris MacNaughton (chris.macnaughton)]

While this is an interesting crashdump, and bit in the logs, especially:

NAME="sdb" MAJ:MIN="8:16" RM="0" SIZE="931.5G" RO="0" TYPE="disk" MOUNTPOINT=""
NAME="sdb1" MAJ:MIN="8:17" RM="0" SIZE="930.5G" RO="0" TYPE="part" MOUNTPOINT=""
NAME="sdb2" MAJ:MIN="8:18" RM="0" SIZE="1G" RO="0" TYPE="part" MOUNTPOINT=""

this bug is marked fix-committed as a result of https://review.openstack.org/#/c/559964 , so you guys should be using openstack-charmers-next/ceph-osd again to work off of master, and could go to stable after the next release where this bug is resolved

[Jason Hobbs (jason-hobbs)]

Chris,

Actually, this fix for this should be backported to stable. Why hasn't it been?

Jason

On Wed, Apr 18, 2018 at 12:59 AM, Chris MacNaughton
<email address hidden> wrote:
> While this is an interesting crashdump, and bit in the logs, especially:
>
> NAME="sdb" MAJ:MIN="8:16" RM="0" SIZE="931.5G" RO="0" TYPE="disk" MOUNTPOINT=""
> NAME="sdb1" MAJ:MIN="8:17" RM="0" SIZE="930.5G" RO="0" TYPE="part" MOUNTPOINT=""
> NAME="sdb2" MAJ:MIN="8:18" RM="0" SIZE="1G" RO="0" TYPE="part" MOUNTPOINT=""
>
> this bug is marked fix-committed as a result of
> https://review.openstack.org/#/c/559964 , so you guys should be using
> openstack-charmers-next/ceph-osd again to work off of master, and could
> go to stable after the next release where this bug is resolved
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1746118
>
> Title:
> Error: Device is mounted
>
> Status in OpenStack ceph-osd charm:
> Fix Committed
>
> Bug description:
> It looks like the ceph-osd charm is trying to initialize my storage
> twice:
>
> 2018-01-29 22:35:21 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> 2018-01-29 22:36:03 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> The second time in fails with an error:
> 2018-01-29 22:36:03 DEBUG mon-relation-changed ceph-disk: Error: Device is mounted: /dev/sdb1
>
> ceph-osd/5 log: http://paste.ubuntu.com/26486128/
>
> bundle: http://paste.ubuntu.com/26486149/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-ceph-osd/+bug/1746118/+subscriptions

[Ryan Beisner (1chb1n)]

This is a tricky backport, as it changes the type of a config option if we do a straight backport. We will have to look further into how/if we can selectively backport a change which is also backward-compatible with current stable config types.

[Chris Gregan (cgregan)]

This issue continues to plague our Pike deployments. The field will continue to deploy Pike on Xenial. Is there a reasonable workaround for this on Pike? Otherwise any deployment going up in the next month will be effected.

[Jason Hobbs (jason-hobbs)]

It seems to me the fix could have been done in a backportable way if
it was planned on being backported. You could introduce a new config
option and deprecate the old one, for example, rather than changing
the existing one.

On Tue, Apr 24, 2018 at 2:11 PM, Chris Gregan
<email address hidden> wrote:
> This issue continues to plague our Pike deployments. The field will
> continue to deploy Pike on Xenial. Is there a reasonable workaround for
> this on Pike? Otherwise any deployment going up in the next month will
> be effected.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1746118
>
> Title:
> Error: Device is mounted
>
> Status in OpenStack ceph-osd charm:
> Fix Committed
>
> Bug description:
> It looks like the ceph-osd charm is trying to initialize my storage
> twice:
>
> 2018-01-29 22:35:21 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> 2018-01-29 22:36:03 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> The second time in fails with an error:
> 2018-01-29 22:36:03 DEBUG mon-relation-changed ceph-disk: Error: Device is mounted: /dev/sdb1
>
> ceph-osd/5 log: http://paste.ubuntu.com/26486128/
>
> bundle: http://paste.ubuntu.com/26486149/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-ceph-osd/+bug/1746118/+subscriptions

[Chris MacNaughton (chris.macnaughton)]

This change, excepting the config change, has been backported in https://review.openstack.org/#/c/562488/ to the stable/18.02 release of the charms on 20 April.

[Chris MacNaughton (chris.macnaughton)]

When is the last time that a failure was experienced on this bug? The fix was made to the charms at master a little over 1 week ago, and backported to the stable charms 6 days ago.

If we have another occurrence of this bug with the fix linked above applied, can we get a new crashdump to evaluate where that is failing?

[Jason Hobbs (jason-hobbs)]

Hey Chris, we hit it again within the last couple of days, but that's
because we were still running from your test branch
(cs:~chris.macnaughton/ceph-osd-7) rather than the -next versions of
the charms. Sorry about that!

On Wed, Apr 25, 2018 at 7:51 AM, Chris MacNaughton
<email address hidden> wrote:
> When is the last time that a failure was experienced on this bug? The
> fix was made to the charms at master a little over 1 week ago, and
> backported to the stable charms 6 days ago.
>
> If we have another occurrence of this bug with the fix linked above
> applied, can we get a new crashdump to evaluate where that is failing?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1746118
>
> Title:
> Error: Device is mounted
>
> Status in OpenStack ceph-osd charm:
> Fix Released
>
> Bug description:
> It looks like the ceph-osd charm is trying to initialize my storage
> twice:
>
> 2018-01-29 22:35:21 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> 2018-01-29 22:36:03 INFO juju-log mon:43: osdize cmd: ['ceph-disk',
> 'prepare', '--fs-type', 'xfs', '--zap-disk', '--filestore',
> '/dev/sdb']
>
> The second time in fails with an error:
> 2018-01-29 22:36:03 DEBUG mon-relation-changed ceph-disk: Error: Device is mounted: /dev/sdb1
>
> ceph-osd/5 log: http://paste.ubuntu.com/26486128/
>
> bundle: http://paste.ubuntu.com/26486149/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charm-ceph-osd/+bug/1746118/+subscriptions