Comment 3 for bug 1830736

Revision history for this message
Yang Liu (yliu12) wrote : Re: Ceph osd process was not recovered after lock and unlock on storage node

This is reproduced in the same lab in WR, the storage node that failed has journal disk configured just fyi.

storage-0:/home/wrsroot# /etc/init.d/ceph status
=== mon.storage-0 ===
{"version":"13.2.2","release":"mimic","release_type":"stable"}
mon.storage-0: running.
=== osd.0 ===
osd.0: not running.
=== osd.1 ===
osd.1: not running.
storage-0:/home/wrsroot#
storage-0:/home/wrsroot# /etc/init.d/ceph stop
=== osd.1 ===
storage-0:/home/wrsroot# /etc/init.d/ceph stop
=== osd.1 ===
Stopping Ceph osd.1 on storage-0...done
=== osd.0 ===
2019-05-30 17:58:23.946 7efc5f5751c0 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected f89591cf-a74f-4cff-85ed-8f93103267e0, invalid (someone else's?) journal
2019-05-30 17:58:23.946 7efc5f5751c0 -1 filestore(/var/lib/ceph/osd/ceph-1) mount(1871): failed to open journal /var/lib/ceph/osd/ceph-1/journal: (22) Invalid argument
2019-05-30 17:58:23.947 7efc5f5751c0 -1 ** ERROR: error flushing journal /var/lib/ceph/osd/ceph-1/journal for object store /var/lib/ceph/osd/ceph-1: (22) Invalid argument
Stopping Ceph osd.0 on storage-0...done
=== mon.storage-0 ===
2019-05-30 17:58:24.105 7f7d57a5e1c0 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected 6a6674e1-3450-4b0e-9618-b414bffff153, invalid (someone else's?) journal
2019-05-30 17:58:24.105 7f7d57a5e1c0 -1 filestore(/var/lib/ceph/osd/ceph-0) mount(1871): failed to open journal /var/lib/ceph/osd/ceph-0/journal: (22) Invalid argument
2019-05-30 17:58:24.105 7f7d57a5e1c0 -1 ** ERROR: error donehing journal /var/lib/ceph/osd/ceph-0/journal for object store /var/lib/ceph/osd/ceph-0: (22) Invalid argument
storage-0:/home/wrsroot# /etc/init.d/ceph stopl 31770...
=== osd.1 ===
Stopping Ceph osd.1 on storage-0...done
=== osd.0 ===
2019-05-30 18:00:15.760 7f70637011c0 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected f89591cf-a74f-4cff-85ed-8f93103267e0, invalid (someone else's?) journal
2019-05-30 18:00:15.760 7f70637011c0 -1 filestore(/var/lib/ceph/osd/ceph-1) mount(1871): failed to open journal /var/lib/ceph/osd/ceph-1/journal: (22) Invalid argument
2019-05-30 18:00:15.761 7f70637011c0 -1 ** ERROR: error flushing journal /var/lib/ceph/osd/ceph-1/journal for object store /var/lib/ceph/osd/ceph-1: (22) Invalid argument
Stopping Ceph osd.0 on storage-0...done
=== mon.storage-0 ===
2019-05-30 18:00:15.923 7fcc881e71c0 -1 journal FileJournal::open: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected 6a6674e1-3450-4b0e-9618-b414bffff153, invalid (someone else's?) journal
2019-05-30 18:00:15.923 7fcc881e71c0 -1 filestore(/var/lib/ceph/osd/ceph-0) mount(1871): failed to open journal /var/lib/ceph/osd/ceph-0/journal: (22) Invalid argument
2019-05-30 18:00:15.923 7fcc881e71c0 -1 ** ERROR: error flushing journal /var/lib/ceph/osd/ceph-0/journal for object store /var/lib/ceph/osd/ceph-0: (22) Invalid argument
Stopping Ceph mon.storage-0 on storage-0...kill 45253...
done