Ceph RBD Mirror Charm

librbd: mirror pool status occasionally leads to segmentation fault

Bug #1820976 reported by Frode Nordahl on 2019-03-20

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Ceph RBD Mirror Charm	Fix Released	Medium	Unassigned
	ceph (Ubuntu)	Fix Released	Medium	Unassigned

Bug Description

2019-03-20 08:55:20 DEBUG juju-log Running _assess_status()
2019-03-20 08:55:23 DEBUG update-status *** Caught signal (Segmentation fault) **
2019-03-20 08:55:23 DEBUG update-status in thread 7f0f0b7fe700 thread_name:tp_librbd
2019-03-20 08:55:23 DEBUG update-status ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)
2019-03-20 08:55:23 DEBUG update-status 1: (()+0x1d00e0) [0x5568590090e0]
2019-03-20 08:55:23 DEBUG update-status 2: (()+0x12890) [0x7f0f26c3c890]
2019-03-20 08:55:23 DEBUG update-status 3: (librados::IoCtxImpl::flush_aio_writes()+0x17) [0x7f0f2fd454c7]
2019-03-20 08:55:23 DEBUG update-status 4: (librados::IoCtx::aio_flush()+0xc) [0x7f0f2fd1d96c]
2019-03-20 08:55:23 DEBUG update-status 5: (()+0x64ad8) [0x7f0f3005fad8]
2019-03-20 08:55:23 DEBUG update-status 6: (()+0x5c298) [0x7f0f30057298]
2019-03-20 08:55:23 DEBUG update-status 7: (()+0x5e079) [0x7f0f30059079]
2019-03-20 08:55:23 DEBUG update-status 8: (()+0x7203d) [0x7f0f3006d03d]
2019-03-20 08:55:23 DEBUG update-status 9: (()+0x72539) [0x7f0f3006d539]
2019-03-20 08:55:23 DEBUG update-status 10: (()+0x5e079) [0x7f0f30059079]
2019-03-20 08:55:23 DEBUG update-status 11: (()+0xff9b9) [0x7f0f300fa9b9]
2019-03-20 08:55:23 DEBUG update-status 12: (()+0xffee8) [0x7f0f300faee8]
2019-03-20 08:55:23 DEBUG update-status 13: (()+0x5e079) [0x7f0f30059079]
2019-03-20 08:55:23 DEBUG update-status 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0x8f7) [0x7f0f27309177]
2019-03-20 08:55:23 DEBUG update-status 15: (ThreadPool::WorkThread::entry()+0x10) [0x7f0f2730a0a0]
2019-03-20 08:55:23 DEBUG update-status 16: (()+0x76db) [0x7f0f26c316db]
2019-03-20 08:55:23 DEBUG update-status 17: (clone()+0x3f) [0x7f0f257ed88f]
2019-03-20 08:55:23 DEBUG update-status 2019-03-20 08:55:23.528 7f0f0b7fe700 -1 *** Caught signal (Segmentation fault) **
2019-03-20 08:55:23 DEBUG update-status in thread 7f0f0b7fe700 thread_name:tp_librbd
2019-03-20 08:55:23 DEBUG update-status
2019-03-20 08:55:23 DEBUG update-status ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)
2019-03-20 08:55:23 DEBUG update-status 1: (()+0x1d00e0) [0x5568590090e0]
2019-03-20 08:55:23 DEBUG update-status 2: (()+0x12890) [0x7f0f26c3c890]
2019-03-20 08:55:23 DEBUG update-status 3: (librados::IoCtxImpl::flush_aio_writes()+0x17) [0x7f0f2fd454c7]
2019-03-20 08:55:23 DEBUG update-status 4: (librados::IoCtx::aio_flush()+0xc) [0x7f0f2fd1d96c]
2019-03-20 08:55:23 DEBUG update-status 5: (()+0x64ad8) [0x7f0f3005fad8]
2019-03-20 08:55:23 DEBUG update-status 6: (()+0x5c298) [0x7f0f30057298]
2019-03-20 08:55:23 DEBUG update-status 7: (()+0x5e079) [0x7f0f30059079]
2019-03-20 08:55:23 DEBUG update-status 8: (()+0x7203d) [0x7f0f3006d03d]
2019-03-20 08:55:23 DEBUG update-status 9: (()+0x72539) [0x7f0f3006d539]
2019-03-20 08:55:23 DEBUG update-status 10: (()+0x5e079) [0x7f0f30059079]
2019-03-20 08:55:23 DEBUG update-status 11: (()+0xff9b9) [0x7f0f300fa9b9]
2019-03-20 08:55:23 DEBUG update-status 12: (()+0xffee8) [0x7f0f300faee8]
2019-03-20 08:55:23 DEBUG update-status 13: (()+0x5e079) [0x7f0f30059079]
2019-03-20 08:55:23 DEBUG update-status 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0x8f7) [0x7f0f27309177]
2019-03-20 08:55:23 DEBUG update-status 15: (ThreadPool::WorkThread::entry()+0x10) [0x7f0f2730a0a0]
2019-03-20 08:55:23 DEBUG update-status 16: (()+0x76db) [0x7f0f26c316db]
2019-03-20 08:55:23 DEBUG update-status 17: (clone()+0x3f) [0x7f0f257ed88f]
2019-03-20 08:55:23 DEBUG update-status NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2019-03-20 08:55:23 DEBUG update-status
2019-03-20 08:55:23 DEBUG update-status --- begin dump of recent events ---
2019-03-20 08:55:23 DEBUG update-status -110> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command perfcounters_dump hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -109> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command 1 hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -108> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command perf dump hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -107> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command perfcounters_schema hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -106> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command perf histogram dump hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -105> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command 2 hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -104> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command perf schema hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -103> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command perf histogram schema hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -102> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command perf reset hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -101> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command config show hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -100> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command config help hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -99> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command config set hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -98> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command config unset hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -97> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command config get hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -96> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command config diff hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -95> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command config diff get hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -94> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command log flush hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -93> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command log dump hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -92> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command log reopen hook 0x55685a4a2380
2019-03-20 08:55:23 DEBUG update-status -91> 2019-03-20 08:55:23.180 7f0f3076ac00 5 asok(0x55685a46a920) register_command dump_mempools hook 0x55685a4d4728
2019-03-20 08:55:23 DEBUG update-status -90> 2019-03-20 08:55:23.192 7f0f3076ac00 10 monclient: get_monmap_and_config
2019-03-20 08:55:23 DEBUG update-status -89> 2019-03-20 08:55:23.192 7f0f3076ac00 10 monclient: build_initial_monmap
2019-03-20 08:55:23 DEBUG update-status -88> 2019-03-20 08:55:23.192 7f0f3076ac00 10 monclient: init
2019-03-20 08:55:23 DEBUG update-status -87> 2019-03-20 08:55:23.192 7f0f3076ac00 5 adding auth protocol: cephx
2019-03-20 08:55:23 DEBUG update-status -86> 2019-03-20 08:55:23.192 7f0f3076ac00 10 monclient: auth_supported 2 method cephx
2019-03-20 08:55:23 DEBUG update-status -85> 2019-03-20 08:55:23.192 7f0f3076ac00 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.rbd-mirror.juju-d7c3c2-12.keyring
2019-03-20 08:55:23 DEBUG update-status -84> 2019-03-20 08:55:23.192 7f0f3076ac00 10 monclient: _reopen_session rank -1
2019-03-20 08:55:23 DEBUG update-status -83> 2019-03-20 08:55:23.192 7f0f3076ac00 10 monclient(hunting): picked mon.noname-b con 0x55685a5a9b70 addr 10.219.3.155:6789/0
2019-03-20 08:55:23 DEBUG update-status -82> 2019-03-20 08:55:23.192 7f0f3076ac00 10 monclient(hunting): picked mon.noname-c con 0x55685a5ac580 addr 10.219.3.29:6789/0
2019-03-20 08:55:23 DEBUG update-status -81> 2019-03-20 08:55:23.192 7f0f3076ac00 10 monclient(hunting): _renew_subs
2019-03-20 08:55:23 DEBUG update-status -80> 2019-03-20 08:55:23.192 7f0f3076ac00 10 monclient(hunting): authenticate will time out at 2019-03-20 09:00:23.198157
2019-03-20 08:55:23 DEBUG update-status -79> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
2019-03-20 08:55:23 DEBUG update-status -78> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient(hunting): got monmap 2, mon.noname-b is now rank -1
2019-03-20 08:55:23 DEBUG update-status -77> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient(hunting): dump:
2019-03-20 08:55:23 DEBUG update-status epoch 2
2019-03-20 08:55:23 DEBUG update-status fsid 9daa41ba-4aea-11e9-8d2e-00163ef81865
2019-03-20 08:55:23 DEBUG update-status last_changed 2019-03-20 08:33:26.053956
2019-03-20 08:55:23 DEBUG update-status created 2019-03-20 08:32:55.946824
2019-03-20 08:55:23 DEBUG update-status 0: 10.219.3.29:6789/0 mon.juju-d7c3c2-1
2019-03-20 08:55:23 DEBUG update-status 1: 10.219.3.137:6789/0 mon.juju-d7c3c2-0
2019-03-20 08:55:23 DEBUG update-status 2: 10.219.3.155:6789/0 mon.juju-d7c3c2-2
2019-03-20 08:55:23 DEBUG update-status
2019-03-20 08:55:23 DEBUG update-status -76> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
2019-03-20 08:55:23 DEBUG update-status -75> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient(hunting): got monmap 2, mon.juju-d7c3c2-1 is now rank 0
2019-03-20 08:55:23 DEBUG update-status -74> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient(hunting): dump:
2019-03-20 08:55:23 DEBUG update-status epoch 2
2019-03-20 08:55:23 DEBUG update-status fsid 9daa41ba-4aea-11e9-8d2e-00163ef81865
2019-03-20 08:55:23 DEBUG update-status last_changed 2019-03-20 08:33:26.053956
2019-03-20 08:55:23 DEBUG update-status created 2019-03-20 08:32:55.946824
2019-03-20 08:55:23 DEBUG update-status 0: 10.219.3.29:6789/0 mon.juju-d7c3c2-1
2019-03-20 08:55:23 DEBUG update-status 1: 10.219.3.137:6789/0 mon.juju-d7c3c2-0
2019-03-20 08:55:23 DEBUG update-status 2: 10.219.3.155:6789/0 mon.juju-d7c3c2-2
2019-03-20 08:55:23 DEBUG update-status
2019-03-20 08:55:23 DEBUG update-status -73> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient(hunting): my global_id is 17885
2019-03-20 08:55:23 DEBUG update-status -72> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient(hunting): my global_id is 7854
2019-03-20 08:55:23 DEBUG update-status -71> 2019-03-20 08:55:23.196 7f0f13fff700 1 monclient: found mon.juju-d7c3c2-1
2019-03-20 08:55:23 DEBUG update-status -70> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient: _send_mon_message to mon.juju-d7c3c2-1 at 10.219.3.29:6789/0
2019-03-20 08:55:23 DEBUG update-status -69> 2019-03-20 08:55:23.196 7f0f3076ac00 5 monclient: authenticate success, global_id 7854
2019-03-20 08:55:23 DEBUG update-status -68> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient: handle_config config(0 keys) v1
2019-03-20 08:55:23 DEBUG update-status -67> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient: handle_monmap mon_map magic: 0 v1
2019-03-20 08:55:23 DEBUG update-status -66> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient: got monmap 2, mon.juju-d7c3c2-1 is now rank 0
2019-03-20 08:55:23 DEBUG update-status -65> 2019-03-20 08:55:23.196 7f0f13fff700 10 monclient: dump:
2019-03-20 08:55:23 DEBUG update-status epoch 2
2019-03-20 08:55:23 DEBUG update-status fsid 9daa41ba-4aea-11e9-8d2e-00163ef81865
2019-03-20 08:55:23 DEBUG update-status last_changed 2019-03-20 08:33:26.053956
2019-03-20 08:55:23 DEBUG update-status created 2019-03-20 08:32:55.946824
2019-03-20 08:55:23 DEBUG update-status 0: 10.219.3.29:6789/0 mon.juju-d7c3c2-1
2019-03-20 08:55:23 DEBUG update-status 1: 10.219.3.137:6789/0 mon.juju-d7c3c2-0
2019-03-20 08:55:23 DEBUG update-status 2: 10.219.3.155:6789/0 mon.juju-d7c3c2-2
2019-03-20 08:55:23 DEBUG update-status
2019-03-20 08:55:23 DEBUG update-status -64> 2019-03-20 08:55:23.196 7f0f127fc700 4 set_mon_vals no callback set
2019-03-20 08:55:23 DEBUG update-status -63> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient: get_monmap_and_config success
2019-03-20 08:55:23 DEBUG update-status -62> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient: shutdown
2019-03-20 08:55:23 DEBUG update-status -61> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient: build_initial_monmap
2019-03-20 08:55:23 DEBUG update-status -60> 2019-03-20 08:55:23.196 7f0f3076ac00 1 librados: starting msgr at -
2019-03-20 08:55:23 DEBUG update-status -59> 2019-03-20 08:55:23.196 7f0f3076ac00 1 librados: starting objecter
2019-03-20 08:55:23 DEBUG update-status -58> 2019-03-20 08:55:23.196 7f0f3076ac00 5 asok(0x55685a46a920) register_command objecter_requests hook 0x55685a5a6bf0
2019-03-20 08:55:23 DEBUG update-status -57> 2019-03-20 08:55:23.196 7f0f3076ac00 1 librados: setting wanted keys
2019-03-20 08:55:23 DEBUG update-status -56> 2019-03-20 08:55:23.196 7f0f3076ac00 1 librados: calling monclient init
2019-03-20 08:55:23 DEBUG update-status -55> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient: init
2019-03-20 08:55:23 DEBUG update-status -54> 2019-03-20 08:55:23.196 7f0f3076ac00 5 adding auth protocol: cephx
2019-03-20 08:55:23 DEBUG update-status -53> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient: auth_supported 2 method cephx
2019-03-20 08:55:23 DEBUG update-status -52> 2019-03-20 08:55:23.196 7f0f3076ac00 2 auth: KeyRing::load: loaded key file /etc/ceph/ceph.client.rbd-mirror.juju-d7c3c2-12.keyring
2019-03-20 08:55:23 DEBUG update-status -51> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient: _reopen_session rank -1
2019-03-20 08:55:23 DEBUG update-status -50> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient(hunting): picked mon.noname-a con 0x55685a5af4c0 addr 10.219.3.137:6789/0
2019-03-20 08:55:23 DEBUG update-status -49> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient(hunting): picked mon.noname-b con 0x55685a5adf00 addr 10.219.3.155:6789/0
2019-03-20 08:55:23 DEBUG update-status -48> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient(hunting): _renew_subs
2019-03-20 08:55:23 DEBUG update-status -47> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient(hunting): authenticate will time out at 2019-03-20 09:00:23.201007
2019-03-20 08:55:23 DEBUG update-status -46> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
2019-03-20 08:55:23 DEBUG update-status -45> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient(hunting): got monmap 2, mon.noname-a is now rank -1
2019-03-20 08:55:23 DEBUG update-status -44> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient(hunting): dump:
2019-03-20 08:55:23 DEBUG update-status epoch 2
2019-03-20 08:55:23 DEBUG update-status fsid 9daa41ba-4aea-11e9-8d2e-00163ef81865
2019-03-20 08:55:23 DEBUG update-status last_changed 2019-03-20 08:33:26.053956
2019-03-20 08:55:23 DEBUG update-status created 2019-03-20 08:32:55.946824
2019-03-20 08:55:23 DEBUG update-status 0: 10.219.3.29:6789/0 mon.juju-d7c3c2-1
2019-03-20 08:55:23 DEBUG update-status 1: 10.219.3.137:6789/0 mon.juju-d7c3c2-0
2019-03-20 08:55:23 DEBUG update-status 2: 10.219.3.155:6789/0 mon.juju-d7c3c2-2
2019-03-20 08:55:23 DEBUG update-status
2019-03-20 08:55:23 DEBUG update-status -43> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient(hunting): my global_id is 7882
2019-03-20 08:55:23 DEBUG update-status -42> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
2019-03-20 08:55:23 DEBUG update-status -41> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient(hunting): got monmap 2, mon.juju-d7c3c2-2 is now rank 2
2019-03-20 08:55:23 DEBUG update-status -40> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient(hunting): dump:
2019-03-20 08:55:23 DEBUG update-status epoch 2
2019-03-20 08:55:23 DEBUG update-status fsid 9daa41ba-4aea-11e9-8d2e-00163ef81865
2019-03-20 08:55:23 DEBUG update-status last_changed 2019-03-20 08:33:26.053956
2019-03-20 08:55:23 DEBUG update-status created 2019-03-20 08:32:55.946824
2019-03-20 08:55:23 DEBUG update-status 0: 10.219.3.29:6789/0 mon.juju-d7c3c2-1
2019-03-20 08:55:23 DEBUG update-status 1: 10.219.3.137:6789/0 mon.juju-d7c3c2-0
2019-03-20 08:55:23 DEBUG update-status 2: 10.219.3.155:6789/0 mon.juju-d7c3c2-2
2019-03-20 08:55:23 DEBUG update-status
2019-03-20 08:55:23 DEBUG update-status -39> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient(hunting): my global_id is 17888
2019-03-20 08:55:23 DEBUG update-status -38> 2019-03-20 08:55:23.196 7f0f137fe700 1 monclient: found mon.juju-d7c3c2-2
2019-03-20 08:55:23 DEBUG update-status -37> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient: _send_mon_message to mon.juju-d7c3c2-2 at 10.219.3.155:6789/0
2019-03-20 08:55:23 DEBUG update-status -36> 2019-03-20 08:55:23.196 7f0f3076ac00 5 monclient: authenticate success, global_id 17888
2019-03-20 08:55:23 DEBUG update-status -35> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient: _renew_subs
2019-03-20 08:55:23 DEBUG update-status -34> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient: _send_mon_message to mon.juju-d7c3c2-2 at 10.219.3.155:6789/0
2019-03-20 08:55:23 DEBUG update-status -33> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient: handle_config config(0 keys) v1
2019-03-20 08:55:23 DEBUG update-status -32> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient: handle_monmap mon_map magic: 0 v1
2019-03-20 08:55:23 DEBUG update-status -31> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient: got monmap 2, mon.juju-d7c3c2-2 is now rank 2
2019-03-20 08:55:23 DEBUG update-status -30> 2019-03-20 08:55:23.196 7f0f137fe700 10 monclient: dump:
2019-03-20 08:55:23 DEBUG update-status epoch 2
2019-03-20 08:55:23 DEBUG update-status fsid 9daa41ba-4aea-11e9-8d2e-00163ef81865
2019-03-20 08:55:23 DEBUG update-status last_changed 2019-03-20 08:33:26.053956
2019-03-20 08:55:23 DEBUG update-status created 2019-03-20 08:32:55.946824
2019-03-20 08:55:23 DEBUG update-status 0: 10.219.3.29:6789/0 mon.juju-d7c3c2-1
2019-03-20 08:55:23 DEBUG update-status 1: 10.219.3.137:6789/0 mon.juju-d7c3c2-0
2019-03-20 08:55:23 DEBUG update-status 2: 10.219.3.155:6789/0 mon.juju-d7c3c2-2
2019-03-20 08:55:23 DEBUG update-status
2019-03-20 08:55:23 DEBUG update-status -29> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient: _renew_subs
2019-03-20 08:55:23 DEBUG update-status -28> 2019-03-20 08:55:23.196 7f0f3076ac00 10 monclient: _send_mon_message to mon.juju-d7c3c2-2 at 10.219.3.155:6789/0
2019-03-20 08:55:23 DEBUG update-status -27> 2019-03-20 08:55:23.196 7f0f11ffb700 4 set_mon_vals no callback set
2019-03-20 08:55:23 DEBUG update-status -26> 2019-03-20 08:55:23.196 7f0f3076ac00 1 librados: init done
2019-03-20 08:55:23 DEBUG update-status -25> 2019-03-20 08:55:23.196 7f0f137fe700 4 mgrc handle_mgr_map Got map version 5
2019-03-20 08:55:23 DEBUG update-status -24> 2019-03-20 08:55:23.196 7f0f137fe700 4 mgrc handle_mgr_map Active mgr is now 10.219.3.29:6800/13921
2019-03-20 08:55:23 DEBUG update-status -23> 2019-03-20 08:55:23.196 7f0f137fe700 4 mgrc reconnect Starting new session with 10.219.3.29:6800/13921
2019-03-20 08:55:23 DEBUG update-status -22> 2019-03-20 08:55:23.200 7f0f3076ac00 5 librbd::io::ImageRequestWQ: 0x55685a42dc70 ImageRequestWQ: ictx=0x55685a5c2010
2019-03-20 08:55:23 DEBUG update-status -21> 2019-03-20 08:55:23.204 7f0f0bfff700 5 asok(0x55685a46a920) register_command rbd cache flush cinder-ceph/volume-7f4f826a-5441-425d-ac46-72902f89b42a hook 0x7f0f000070a0
2019-03-20 08:55:23 DEBUG update-status -20> 2019-03-20 08:55:23.204 7f0f0bfff700 5 asok(0x55685a46a920) register_command rbd cache invalidate cinder-ceph/volume-7f4f826a-5441-425d-ac46-72902f89b42a hook 0x7f0f000070a0
2019-03-20 08:55:23 DEBUG update-status -19> 2019-03-20 08:55:23.232 7f0f0bfff700 5 librbd::io::ImageRequestWQ: 0x55685a42dc70 block_writes: 0x55685a5c2010, num=1
2019-03-20 08:55:23 DEBUG update-status -18> 2019-03-20 08:55:23.232 7f0f0b7fe700 5 librbd::cache::ObjectCacherObjectDispatch: 0x7f0f00006030 init:
2019-03-20 08:55:23 DEBUG update-status -17> 2019-03-20 08:55:23.232 7f0f0b7fe700 5 librbd::cache::ObjectCacherObjectDispatch: 0x7f0f00006030 init: enabling caching...
2019-03-20 08:55:23 DEBUG update-status -16> 2019-03-20 08:55:23.232 7f0f0b7fe700 5 librbd::cache::ObjectCacherObjectDispatch: 0x7f0f00006030 init: Initial cache settings: size=33554432 num_objects=10 max_dirty=25165824 target_dirty=16777216 max_dirty_age=1
2019-03-20 08:55:23 DEBUG update-status -15> 2019-03-20 08:55:23.232 7f0f0b7fe700 5 librbd::cache::ObjectCacherObjectDispatch: 0x7f0f00006030 init: cache bytes 33554432 -> about 855 objects
2019-03-20 08:55:23 DEBUG update-status -14> 2019-03-20 08:55:23.232 7f0f0b7fe700 5 librbd::io::ObjectDispatcher: 0x55685a456ca0 register_object_dispatch: object_dispatch_layer=1
2019-03-20 08:55:23 DEBUG update-status -13> 2019-03-20 08:55:23.356 7f0efffff700 1 Journaler::thread_pool worker finish
2019-03-20 08:55:23 DEBUG update-status -12> 2019-03-20 08:55:23.356 7f0f0bfff700 5 librbd::Watcher: 0x7f0f00004340 block_notifies: blocked_count=1
2019-03-20 08:55:23 DEBUG update-status -11> 2019-03-20 08:55:23.356 7f0f0b7fe700 5 librbd::io::ImageRequestWQ: 0x55685a42dc70 shut_down: shut_down: in_flight=0
2019-03-20 08:55:23 DEBUG update-status -10> 2019-03-20 08:55:23.356 7f0f0b7fe700 5 librbd::io::ImageRequestWQ: 0x55685a42dc70 unblock_writes: 0x55685a5c2010, num=0
2019-03-20 08:55:23 DEBUG update-status -9> 2019-03-20 08:55:23.520 7f0f0b7fe700 5 librbd::io::ObjectDispatcher: 0x55685a456ca0 shut_down:
2019-03-20 08:55:23 DEBUG update-status -8> 2019-03-20 08:55:23.520 7f0f0b7fe700 5 librbd::io::ObjectDispatch: 0x55685a5b9850 shut_down:
2019-03-20 08:55:23 DEBUG update-status -7> 2019-03-20 08:55:23.520 7f0f0b7fe700 5 librbd::cache::ObjectCacherObjectDispatch: 0x7f0f00006030 shut_down:
2019-03-20 08:55:23 DEBUG update-status -6> 2019-03-20 08:55:23.520 7f0f0b7fe700 5 asok(0x55685a46a920) unregister_command rbd cache flush cinder-ceph/volume-7f4f826a-5441-425d-ac46-72902f89b42a
2019-03-20 08:55:23 DEBUG update-status -5> 2019-03-20 08:55:23.520 7f0f0b7fe700 5 asok(0x55685a46a920) unregister_command rbd cache invalidate cinder-ceph/volume-7f4f826a-5441-425d-ac46-72902f89b42a
2019-03-20 08:55:23 DEBUG update-status -4> 2019-03-20 08:55:23.520 7f0f3076ac00 5 asok(0x55685a46a920) unregister_command objecter_requests
2019-03-20 08:55:23 DEBUG update-status -3> 2019-03-20 08:55:23.520 7f0f3076ac00 10 monclient: shutdown
2019-03-20 08:55:23 DEBUG update-status -2> 2019-03-20 08:55:23.524 7f0f3076ac00 1 librados: shutdown
2019-03-20 08:55:23 DEBUG update-status -1> 2019-03-20 08:55:23.524 7f0f3076ac00 5 asok(0x55685a46a920) unregister_command dump_mempools
2019-03-20 08:55:23 DEBUG update-status 0> 2019-03-20 08:55:23.528 7f0f0b7fe700 -1 *** Caught signal (Segmentation fault) **
2019-03-20 08:55:23 DEBUG update-status in thread 7f0f0b7fe700 thread_name:tp_librbd
2019-03-20 08:55:23 DEBUG update-status
2019-03-20 08:55:23 DEBUG update-status ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable)
2019-03-20 08:55:23 DEBUG update-status 1: (()+0x1d00e0) [0x5568590090e0]
2019-03-20 08:55:23 DEBUG update-status 2: (()+0x12890) [0x7f0f26c3c890]
2019-03-20 08:55:23 DEBUG update-status 3: (librados::IoCtxImpl::flush_aio_writes()+0x17) [0x7f0f2fd454c7]
2019-03-20 08:55:23 DEBUG update-status 4: (librados::IoCtx::aio_flush()+0xc) [0x7f0f2fd1d96c]
2019-03-20 08:55:23 DEBUG update-status 5: (()+0x64ad8) [0x7f0f3005fad8]
2019-03-20 08:55:23 DEBUG update-status 6: (()+0x5c298) [0x7f0f30057298]
2019-03-20 08:55:23 DEBUG update-status 7: (()+0x5e079) [0x7f0f30059079]
2019-03-20 08:55:23 DEBUG update-status 8: (()+0x7203d) [0x7f0f3006d03d]
2019-03-20 08:55:23 DEBUG update-status 9: (()+0x72539) [0x7f0f3006d539]
2019-03-20 08:55:23 DEBUG update-status 10: (()+0x5e079) [0x7f0f30059079]
2019-03-20 08:55:23 DEBUG update-status 11: (()+0xff9b9) [0x7f0f300fa9b9]
2019-03-20 08:55:23 DEBUG update-status 12: (()+0xffee8) [0x7f0f300faee8]
2019-03-20 08:55:23 DEBUG update-status 13: (()+0x5e079) [0x7f0f30059079]
2019-03-20 08:55:23 DEBUG update-status 14: (ThreadPool::worker(ThreadPool::WorkThread*)+0x8f7) [0x7f0f27309177]
2019-03-20 08:55:23 DEBUG update-status 15: (ThreadPool::WorkThread::entry()+0x10) [0x7f0f2730a0a0]
2019-03-20 08:55:23 DEBUG update-status 16: (()+0x76db) [0x7f0f26c316db]
2019-03-20 08:55:23 DEBUG update-status 17: (clone()+0x3f) [0x7f0f257ed88f]
2019-03-20 08:55:23 DEBUG update-status NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2019-03-20 08:55:23 DEBUG update-status
2019-03-20 08:55:23 DEBUG update-status --- logging levels ---
2019-03-20 08:55:23 DEBUG update-status 0/ 5 none
2019-03-20 08:55:23 DEBUG update-status 0/ 1 lockdep
2019-03-20 08:55:23 DEBUG update-status 0/ 1 context
2019-03-20 08:55:23 DEBUG update-status 1/ 1 crush
2019-03-20 08:55:23 DEBUG update-status 1/ 5 mds
2019-03-20 08:55:23 DEBUG update-status 1/ 5 mds_balancer
2019-03-20 08:55:23 DEBUG update-status 1/ 5 mds_locker
2019-03-20 08:55:23 DEBUG update-status 1/ 5 mds_log
2019-03-20 08:55:23 DEBUG update-status 1/ 5 mds_log_expire
2019-03-20 08:55:23 DEBUG update-status 1/ 5 mds_migrator
2019-03-20 08:55:23 DEBUG update-status 0/ 1 buffer
2019-03-20 08:55:23 DEBUG update-status 0/ 1 timer
2019-03-20 08:55:23 DEBUG update-status 0/ 1 filer
2019-03-20 08:55:23 DEBUG update-status 0/ 1 striper
2019-03-20 08:55:23 DEBUG update-status 0/ 1 objecter
2019-03-20 08:55:23 DEBUG update-status 0/ 5 rados
2019-03-20 08:55:23 DEBUG update-status 0/ 5 rbd
2019-03-20 08:55:23 DEBUG update-status 0/ 5 rbd_mirror
2019-03-20 08:55:23 DEBUG update-status 0/ 5 rbd_replay
2019-03-20 08:55:23 DEBUG update-status 0/ 5 journaler
2019-03-20 08:55:23 DEBUG update-status 0/ 5 objectcacher
2019-03-20 08:55:23 DEBUG update-status 0/ 5 client
2019-03-20 08:55:23 DEBUG update-status 1/ 5 osd
2019-03-20 08:55:23 DEBUG update-status 0/ 5 optracker
2019-03-20 08:55:23 DEBUG update-status 0/ 5 objclass
2019-03-20 08:55:23 DEBUG update-status 1/ 3 filestore
2019-03-20 08:55:23 DEBUG update-status 1/ 3 journal
2019-03-20 08:55:23 DEBUG update-status 0/ 0 ms
2019-03-20 08:55:23 DEBUG update-status 1/ 5 mon
2019-03-20 08:55:23 DEBUG update-status 0/10 monc
2019-03-20 08:55:23 DEBUG update-status 1/ 5 paxos
2019-03-20 08:55:23 DEBUG update-status 0/ 5 tp
2019-03-20 08:55:23 DEBUG update-status 1/ 5 auth
2019-03-20 08:55:23 DEBUG update-status 1/ 5 crypto
2019-03-20 08:55:23 DEBUG update-status 1/ 1 finisher
2019-03-20 08:55:23 DEBUG update-status 1/ 1 reserver
2019-03-20 08:55:23 DEBUG update-status 1/ 5 heartbeatmap
2019-03-20 08:55:23 DEBUG update-status 1/ 5 perfcounter
2019-03-20 08:55:23 DEBUG update-status 1/ 5 rgw
2019-03-20 08:55:23 DEBUG update-status 1/ 5 rgw_sync
2019-03-20 08:55:23 DEBUG update-status 1/10 civetweb
2019-03-20 08:55:23 DEBUG update-status 1/ 5 javaclient
2019-03-20 08:55:23 DEBUG update-status 1/ 5 asok
2019-03-20 08:55:23 DEBUG update-status 1/ 1 throttle
2019-03-20 08:55:23 DEBUG update-status 0/ 0 refs
2019-03-20 08:55:23 DEBUG update-status 1/ 5 xio
2019-03-20 08:55:23 DEBUG update-status 1/ 5 compressor
2019-03-20 08:55:23 DEBUG update-status 1/ 5 bluestore
2019-03-20 08:55:23 DEBUG update-status 1/ 5 bluefs
2019-03-20 08:55:23 DEBUG update-status 1/ 3 bdev
2019-03-20 08:55:23 DEBUG update-status 1/ 5 kstore
2019-03-20 08:55:23 DEBUG update-status 4/ 5 rocksdb
2019-03-20 08:55:23 DEBUG update-status 4/ 5 leveldb
2019-03-20 08:55:23 DEBUG update-status 4/ 5 memdb
2019-03-20 08:55:23 DEBUG update-status 1/ 5 kinetic
2019-03-20 08:55:23 DEBUG update-status 1/ 5 fuse
2019-03-20 08:55:23 DEBUG update-status 1/ 5 mgr
2019-03-20 08:55:23 DEBUG update-status 1/ 5 mgrc
2019-03-20 08:55:23 DEBUG update-status 1/ 5 dpdk
2019-03-20 08:55:23 DEBUG update-status 1/ 5 eventtrace
2019-03-20 08:55:23 DEBUG update-status -2/-2 (syslog threshold)
2019-03-20 08:55:23 DEBUG update-status 99/99 (stderr threshold)
2019-03-20 08:55:23 DEBUG update-status max_recent 500
2019-03-20 08:55:23 DEBUG update-status max_new 1000
2019-03-20 08:55:23 DEBUG update-status log_file
2019-03-20 08:55:23 DEBUG update-status --- end dump of recent events ---
2019-03-20 08:55:27 DEBUG update-status Traceback (most recent call last):
2019-03-20 08:55:27 DEBUG update-status File "/var/lib/juju/agents/unit-ceph-rbd-mirror-0/charm/hooks/update-status", line 22, in <module>
2019-03-20 08:55:27 DEBUG update-status main()
2019-03-20 08:55:27 DEBUG update-status File "/var/lib/juju/agents/unit-ceph-rbd-mirror-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 83, in main
2019-03-20 08:55:27 DEBUG update-status hookenv._run_atexit()
2019-03-20 08:55:27 DEBUG update-status File "/var/lib/juju/agents/unit-ceph-rbd-mirror-0/.venv/lib/python3.6/site-packages/charmhelpers/core/hookenv.py", line 1220, in _run_atexit
2019-03-20 08:55:27 DEBUG update-status callback(*args, **kwargs)
2019-03-20 08:55:27 DEBUG update-status File "/var/lib/juju/agents/unit-ceph-rbd-mirror-0/.venv/lib/python3.6/site-packages/charms_openstack/charm/core.py", line 1113, in atexit_assess_status
2019-03-20 08:55:27 DEBUG update-status self._assess_status()
2019-03-20 08:55:27 DEBUG update-status File "/var/lib/juju/agents/unit-ceph-rbd-mirror-0/.venv/lib/python3.6/site-packages/charms_openstack/charm/core.py", line 1095, in _assess_status
2019-03-20 08:55:27 DEBUG update-status state, message = f()
2019-03-20 08:55:27 DEBUG update-status File "lib/charm/openstack/ceph_rbd_mirror.py", line 69, in custom_assess_status_check
2019-03-20 08:55:27 DEBUG update-status (pool for pool, attrs in endpoint.pools.items()
2019-03-20 08:55:27 DEBUG update-status File "lib/charm/openstack/ceph_rbd_mirror.py", line 129, in mirror_pools_summary
2019-03-20 08:55:27 DEBUG update-status pool_stat = self.mirror_pool_status(pool)
2019-03-20 08:55:27 DEBUG update-status File "lib/charm/openstack/ceph_rbd_mirror.py", line 121, in mirror_pool_status
2019-03-20 08:55:27 DEBUG update-status universal_newlines=True)
2019-03-20 08:55:27 DEBUG update-status File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
2019-03-20 08:55:27 DEBUG update-status **kwargs).stdout
2019-03-20 08:55:27 DEBUG update-status File "/usr/lib/python3.6/subprocess.py", line 418, in run
2019-03-20 08:55:27 DEBUG update-status output=stdout, stderr=stderr)
2019-03-20 08:55:27 DEBUG update-status subprocess.CalledProcessError: Command '['rbd', '--id', 'rbd-mirror.juju-d7c3c2-12', 'mirror', 'pool', 'status', '--format', 'json', '--verbose', 'cinder-ceph']' died with <Signals.SIGSEGV: 11>.

Revision history for this message

Frode Nordahl (fnordahl) wrote on 2019-03-29:

https://review.openstack.org/#/c/648618/

Changed in charm-ceph-rbd-mirror:
status:	New → In Progress
importance:	Undecided → Medium
assignee:	nobody → Frode Nordahl (fnordahl)

Revision history for this message

Jason Dillaman (jdillaman) wrote on 2019-03-29:

Upstream ticket: http://tracker.ceph.com/issues/39031

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-03: Fix merged to charm-ceph-rbd-mirror (master)

Reviewed: https://review.openstack.org/648618
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-rbd-mirror/commit/?id=cc6897b738d65ec21d10ad596be4138e96990211
Submitter: Zuul
Branch: master

commit cc6897b738d65ec21d10ad596be4138e96990211
Author: Frode Nordahl <email address hidden>
Date: Fri Mar 29 07:23:01 2019 +0100

Handle inability to retrieve status gracefully

Tactical fix for occational segfault in ``rbd mirror pool status``

Change-Id: Ic50d06ee646ca525085fa009d344c8eabd178839
Partial-Bug: #1820976

Revision history for this message

James Page (james-page) wrote on 2019-07-03:

Backports are inflight upstream, will be picked up in the next round of stable updates.

Changed in ceph (Ubuntu):
status:	New → Triaged
importance:	Undecided → Medium

Revision history for this message

James Page (james-page) wrote on 2020-04-09:

The ceph updates in proposed pockets include the backported fixes for this issue.

Changed in ceph (Ubuntu):
status:	Triaged → Fix Committed

Revision history for this message

James Page (james-page) wrote on 2020-04-16:

SRU updates for 13.2.8 and 14.2.8 have been released so marking this as 'Fix Released'

Changed in ceph (Ubuntu):
status:	Fix Committed → Fix Released

Frode Nordahl (fnordahl) on 2020-09-17

Changed in charm-ceph-rbd-mirror:
status:	In Progress → Fix Released
assignee:	Frode Nordahl (fnordahl) → nobody

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

auto-tracker.ceph.com #39031 Edit

Bug watches keep track of this bug in other bug trackers.