Bug #1451947 “AP Failed: Add unit to 'ceph' service on 'node4.ma...” : Bugs : ceph package : Juju Charms Collection

Revision history for this message

mahmoh (mahmoh) wrote on 2015-05-05:

#1

all-machines.log Edit (2.1 MiB, text/plain)
api.log Edit (103.1 KiB, text/plain)
appserver.log Edit (60.1 KiB, text/plain)
async-frontend.log Edit (111.7 KiB, text/plain)
combo-loader.log Edit (109.0 KiB, text/plain)
job-handler.log Edit (82.8 KiB, text/plain)
juju-status.out Edit (15.3 KiB, text/plain)
juju-sync.log Edit (29.3 KiB, text/plain)
landscape-profiles.log Edit (4.0 KiB, text/plain)
landscape-setup.log Edit (2.1 KiB, text/plain)
maas-node-node0vm0.xml Edit (19.5 KiB, text/plain)
maas-node-node0vm1.xml Edit (19.5 KiB, text/plain)
maas-node-node1.xml Edit (32.8 KiB, text/plain)
maas-node-node3.xml Edit (32.8 KiB, text/plain)
maas-node-node4.xml Edit (31.7 KiB, text/plain)
maas-node-node5.xml Edit (31.0 KiB, text/plain)
package-search.log Edit (394 bytes, text/plain)
package-upload.log Edit (40.2 KiB, text/plain)
pingserver.log Edit (84.7 KiB, text/plain)
process-alerts.log Edit (972 bytes, text/plain)
update-alerts.log Edit (3.8 KiB, text/plain)

Revision history for this message

David Britton (dpb) wrote on 2015-05-05:

#2

Download full text (5.1 KiB)

http://paste.ubuntu.com/10991033/

unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 #007Caution: invalid backup GPT header, but valid main header; regenerating
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 backup header from main header.
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 on the recovery & transformation menu to examine the two tables.
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 Warning! One or more CRCs don't match. You should repair the disk!
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 #015Building dependency tree... 0%#015#015Building dependency tree... 0%#015#015Building dependency tree... 50%#015#015Building dependency tree... 50%#015#015Building dependency tree
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 #015Reading state information... 0%#015#015Reading state information... 0%#015#015Reading state information... Done
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 Traceback (most recent call last):
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/usr/sbin/ceph-disk", line 2768, in <module>
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 main()
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/usr/sbin/ceph-disk", line 2746, in main
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 args.func(args)
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/usr/sbin/ceph-disk", line 1390, in main_prepare
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 zap(args.data)
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 File "/usr/sbin/ceph-disk", line 1013, in zap
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 with file(dev, 'wb') as dev_file:
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 IOError: [Errno 123] No medium found: '/dev/sdb'
unit-ceph-1[32533]: 2015-05-05 17:16:24 ERROR unit.ceph/1.juju-log cmd.go:247 mon:3: Unable to initialize device: /dev/sdb
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 Traceback (most recent call last):
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40...

http://paste.ubuntu.com/10991033/

unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 #007Caution: invalid backup GPT header, but valid main header; regenerating
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 backup header from main header.
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 on the recovery & transformation menu to examine the two tables.
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 Warning! One or more CRCs don't match. You should repair the disk!
unit-ceph-2[32498]: 2015-05-05 17:16:23 INFO unit.ceph/2.mon-relation-changed logger.go:40 
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 #015Building dependency tree... 0%#015#015Building dependency tree... 0%#015#015Building dependency tree... 50%#015#015Building dependency tree... 50%#015#015Building dependency tree       
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 #015Reading state information... 0%#015#015Reading state information... 0%#015#015Reading state information... Done
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 Traceback (most recent call last):
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40   File "/usr/sbin/ceph-disk", line 2768, in <module>
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40     main()
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40   File "/usr/sbin/ceph-disk", line 2746, in main
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40     args.func(args)
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40   File "/usr/sbin/ceph-disk", line 1390, in main_prepare
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40     zap(args.data)
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40   File "/usr/sbin/ceph-disk", line 1013, in zap
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40     with file(dev, 'wb') as dev_file:
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 IOError: [Errno 123] No medium found: '/dev/sdb'
unit-ceph-1[32533]: 2015-05-05 17:16:24 ERROR unit.ceph/1.juju-log cmd.go:247 mon:3: Unable to initialize device: /dev/sdb
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 Traceback (most recent call last):
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40   File "/var/lib/juju/agents/unit-ceph-1/charm/hooks/mon-relation-changed", line 312, in <module>
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40     hooks.execute(sys.argv)
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40   File "/var/lib/juju/agents/unit-ceph-1/charm/hooks/charmhelpers/core/hookenv.py", line 508, in execute
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40     self._hooks[hook_name]()
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40   File "/var/lib/juju/agents/unit-ceph-1/charm/hooks/mon-relation-changed", line 196, in mon_relation
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40     reformat_osd(), config('ignore-device-errors'))
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40   File "/var/lib/juju/agents/unit-ceph-1/charm/hooks/ceph.py", line 327, in osdize
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40     osdize_dev(dev, osd_format, osd_journal, reformat_osd, ignore_errors)
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40   File "/var/lib/juju/agents/unit-ceph-1/charm/hooks/ceph.py", line 375, in osdize_dev
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40     raise e
unit-ceph-1[32533]: 2015-05-05 17:16:24 INFO unit.ceph/1.mon-relation-changed logger.go:40 subprocess.CalledProcessError: Command '['ceph-disk-prepare', '--fs-type', u'xfs', '--zap-disk', u'/dev/sdb']' returned non-zero exit status 1
unit-ceph-1[32533]: 2015-05-05 17:16:24 ERROR juju.worker.uniter.operation runhook.go:86 hook "mon-relation-changed" failed: exit status 1

This looks like the operative failure.  I'd suggest looking into possible hardware failures on that disk.  I'll add ceph on here so perhaps the maintainers of that project could have a chance to sound in.

information type:

Proprietary → Public

Revision history for this message

Dean Henrichsmeyer (dean) wrote on 2015-05-05:

#3

This can happen with a dead disk as well. I've seen this when USB devices that are used as second disks die.

Dean Henrichsmeyer (dean) on 2015-05-11

no longer affects:

landscape

Juju Charms Collection
ceph package

AP Failed: Add unit to 'ceph' service on 'node4.maas'

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Juju Charms Collectionceph package

AP Failed: Add unit to 'ceph' service on 'node4.maas'

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Juju Charms Collection
ceph package