Fuel for OpenStack

  1. Bug #1431912
  2. Activity log

Activity log for bug #1431912

Date Who What changed Old value New value Message
2015-03-13 14:57:28 krogon-intel bug added bug
2015-03-13 15:00:02 krogon-intel attachment added node-425.log https://bugs.launchpad.net/fuel/+bug/1431912/+attachment/4344388/+files/node-425.log
2015-03-13 15:00:32 krogon-intel description Fuel 6.0. Deploying "Juno on CentOS 6.5". During /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare] it was able to set only first journal, because rest OSD were already monted. Details: - sda SSD: base system - sdb SSD: journal (sdb4-sdb15 10GB partitions created by fuel) - sdc-sdn HDD: ceph [root@node-425 ~]# ls -l /var/lib/ceph/osd/*/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-32/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-37/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-41/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-44/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-48/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-53/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-57/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-61/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-64/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-69/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-72/journal lrwxrwxrwx 1 root root 10 Mar 13 09:38 /var/lib/ceph/osd/ceph-78/journal -> /dev/sdb10 Part of ceph.log [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.9): /usr/bin/ceph-deploy osd prepare node-425:/dev/sdc4:/dev/sdb10 node-425:/dev/sdd4:/dev/sdb11 node-425:/dev/sde4:/dev/sdb12 node-425:/dev/sdf4:/dev/sdb13 node-425:/dev/sdg4:/dev/sdb14 node-425:/dev/sdh4:/dev/sdb15 node-425:/dev/sdi4:/dev/sdb4 node-425:/dev/sdj4:/dev/sdb5 node-425:/dev/sdk4:/dev/sdb6 node-425:/dev/sdl4:/dev/sdb7 node-425:/dev/sdm4:/dev/sdb8 node-425:/dev/sdn4:/dev/sdb9 [ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node-425:/dev/sdc4:/dev/sdb10 node-425:/dev/sdd4:/dev/sdb11 node-425:/dev/sde4:/dev/sdb12 node-425:/dev/sdf4:/dev/sdb13 node-425:/dev/sdg4:/dev/sdb14 node-425:/dev/sdh4:/dev/sdb15 node-425:/dev/sdi4:/dev/sdb4 node-425:/dev/sdj4:/dev/sdb5 node-425:/dev/sdk4:/dev/sdb6 node-425:/dev/sdl4:/dev/sdb7 node-425:/dev/sdm4:/dev/sdb8 node-425:/dev/sdn4:/dev/sdb9 [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Deploying osd to node-425 [node-425][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [node-425][WARNING] osd keyring does not exist yet, creating one [node-425][DEBUG ] create a keyring file [node-425][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sdc4 journal /dev/sdb10 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdc4 /dev/sdb10 [node-425][DEBUG ] meta-data=/dev/sdc4 isize=2048 agcount=4, agsize=244152512 blks [node-425][DEBUG ] = sectsz=512 attr=2, projid32bit=0 [node-425][DEBUG ] data = bsize=4096 blocks=976610048, imaxpct=5 [node-425][DEBUG ] = sunit=0 swidth=0 blks [node-425][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0 [node-425][DEBUG ] log =internal log bsize=4096 blocks=476860, version=2 [node-425][DEBUG ] = sectsz=512 sunit=0 blks, lazy-count=1 [node-425][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0 [node-425][WARNING] WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the same device as the osd data [node-425][WARNING] last arg is not the whole disk [node-425][WARNING] call: partx -opts device wholedisk [node-425][INFO ] checking OSD status... [node-425][INFO ] Running command: ceph --cluster=ceph osd stat --format=json [node-425][WARNING] there are 4 OSDs down [node-425][WARNING] there are 4 OSDs out [ceph_deploy.osd][DEBUG ] Host node-425 is now ready for osd use. [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sdd4 journal /dev/sdb11 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdd4 /dev/sdb11 [node-425][WARNING] ceph-disk: Error: Device is mounted: /dev/sdd4 [node-425][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdd4 /dev/sdb11 [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sde4 journal /dev/sdb12 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sde4 /dev/sdb12 [node-425][WARNING] ceph-disk: Error: Device is mounted: /dev/sde4 [node-425][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sde4 /dev/sdb12 [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sdf4 journal /dev/sdb13 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdf4 /dev/sdb13 [node-425][WARNING] ceph-disk: Error: Device is mounted: /dev/sdf4 [node-425][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdf4 /dev/sdb13 [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sdg4 journal /dev/sdb14 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdg4 /dev/sdb14 [node-425][WARNING] ceph-disk: Error: Device is mounted: /dev/sdg4 [node-425][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdg4 /dev/sdb14 [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sdh4 journal /dev/sdb15 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdh4 /dev/sdb15 [node-425][WARNING] ceph-disk: Error: Device is mounted: /dev/sdh4 [node-425][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdh4 /dev/sdb15 [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sdi4 journal /dev/sdb4 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdi4 /dev/sdb4 [node-425][WARNING] ceph-disk: Error: Device is mounted: /dev/sdi4 [node-425][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdi4 /dev/sdb4 [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sdj4 journal /dev/sdb5 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdj4 /dev/sdb5 [node-425][WARNING] ceph-disk: Error: Device is mounted: /dev/sdj4 [node-425][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdj4 /dev/sdb5 [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sdk4 journal /dev/sdb6 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdk4 /dev/sdb6 [node-425][WARNING] ceph-disk: Error: Device is mounted: /dev/sdk4 [node-425][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdk4 /dev/sdb6 [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sdl4 journal /dev/sdb7 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdl4 /dev/sdb7 [node-425][WARNING] ceph-disk: Error: Device is mounted: /dev/sdl4 [node-425][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdl4 /dev/sdb7 [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sdm4 journal /dev/sdb8 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdm4 /dev/sdb8 [node-425][WARNING] ceph-disk: Error: Device is mounted: /dev/sdm4 [node-425][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdm4 /dev/sdb8 [node-425][DEBUG ] connected to host: node-425 [node-425][DEBUG ] detect platform information from remote host [node-425][DEBUG ] detect machine type [ceph_deploy.osd][INFO ] Distro info: CentOS 6.5 Final [ceph_deploy.osd][DEBUG ] Preparing host node-425 disk /dev/sdn4 journal /dev/sdb9 activate False [node-425][INFO ] Running command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdn4 /dev/sdb9 [node-425][WARNING] ceph-disk: Error: Device is mounted: /dev/sdn4 [node-425][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: ceph-disk-prepare --fs-type xfs --cluster ceph -- /dev/sdn4 /dev/sdb9 [ceph_deploy][ERROR ] GenericError: Failed to create 11 OSDs Fuel 6.0. Deploying "Juno on CentOS 6.5". During /Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare] it was able to set only first journal, because rest OSD were already monted. Details: - sda SSD: base system - sdb SSD: journal (sdb4-sdb15 10GB partitions created by fuel) - sdc-sdn HDD: ceph [root@node-425 ~]# ls -l /var/lib/ceph/osd/*/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-32/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-37/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-41/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-44/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-48/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-53/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-57/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-61/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-64/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-69/journal -rw-r--r-- 1 root root 2147483648 Mar 13 09:40 /var/lib/ceph/osd/ceph-72/journal lrwxrwxrwx 1 root root 10 Mar 13 09:38 /var/lib/ceph/osd/ceph-78/journal -> /dev/sdb10 ceph.log in attachment
2015-03-17 00:32:24 Ryan Moe fuel: status New Incomplete
2015-03-17 21:50:13 Vladimir Kuklin fuel: milestone 6.1
2015-03-23 08:53:20 Vladimir Kuklin fuel: status Incomplete Confirmed
2015-03-23 08:53:22 Vladimir Kuklin fuel: importance Undecided High
2015-03-23 08:53:30 Vladimir Kuklin fuel: assignee Fuel Library Team (fuel-library)
2015-03-23 08:53:35 Vladimir Kuklin tags customer-found
2015-03-27 11:40:33 Vladimir Kuklin fuel: status Confirmed Incomplete
2015-04-03 23:08:39 Fuel Devops McRobotson fuel/6.0.x: milestone 6.0.1
2015-04-03 23:08:39 Fuel Devops McRobotson fuel/6.0.x: assignee Fuel Library Team (fuel-library)
2015-04-06 11:17:35 Vladimir Kuklin fuel/6.0.x: status New Incomplete
2015-04-27 17:52:18 Vladimir Kuklin fuel: status Incomplete Invalid
2015-04-27 17:52:21 Vladimir Kuklin fuel/6.0.x: status Incomplete Invalid