Environment deployment failed with ceph-deploy 1.5.2

Bug #1333814 reported by Alexander Podrepniy
36
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Kostiantyn Danylov
5.0.x
Invalid
Critical
Sergii Golovatiuk

Bug Description

Environment deployment failed on Fuel v5.1 build 269 with ceph-deploy-1.5.2-0.mira1
Deployment failed on both enviroments with errors.

Fuel:
"build_id": "2014-06-24_14-57-29",
"mirantis": "yes",
"build_number": "269",
"ostf_sha": "4d2efa822344b6ca022ec4086b6f083c07d90e14",
"nailgun_sha": "b8713a4026239e7b1769e64239e54950ef7c739b",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "b8c33af2da36f81828e68b77da4025e59aee3283",
"astute_sha": "694b5a55695e01e1c42185bfac9cc7a641a9bd48",
"release": "5.1",
"fuellib_sha": "8ec900a2bc96a463a72405a674e41f52fc0d3e14"

Environment 1:
OS: CentOS
Mode: Multi-node with HA (1 controller node)
Storage: Cinder Ceph and Glance Ceph
Nodes: 1 Controller, 1 Compute, 2 Ceph OSD

Puppet log from Controller node:
=============================
2014-06-24 15:32:25 ERR (/Stage[main]/Ceph::Mon/Exec[ceph-deploy mon create]/returns) change from notrun to 0 failed: ceph-deploy mon create node-1:192.168.0.3 returned 1 instead of one of [0]
(useless ruby stack trace removed)
=============================

Environment 2:
OS: CentOS
Mode: Multi-node
Storage: Cinder Ceph and Glance Ceph
Nodes: 1 Controller, 1 Compute, 2 Ceph OSD

Puppet log from Controller node for this environment is the same.

(update)

The root cause of this bug is ceph-deploy failing due to mismatching ceph.conf in /root and /etc/ceph (indicative error message is "config file /etc/ceph/ceph.conf exists with different content"). Please do not reopen due to any other failure of ceph-deploy, they will have a different root cause.

Tags: ceph
Revision history for this message
Alexander Podrepniy (apodrepniy) wrote :
Ryan Moe (rmoe)
Changed in fuel:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Andrew Woodward (xarses)
Changed in fuel:
milestone: none → 5.1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/102381

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/102381
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=eecd91cde59483e30397342cdf5ab7e77d95d9ac
Submitter: Jenkins
Branch: master

commit eecd91cde59483e30397342cdf5ab7e77d95d9ac
Author: Andrew Woodward <email address hidden>
Date: Tue Jun 24 16:01:00 2014 -0700

    Add link to /root/.cephdeploy.conf -> /etc/ceph/ceph.conf

    ceph-deploy requires that its local config file (previously $(pwd)/ceph.conf)
    is exactly the same as /etc/ceph/ceph.conf on the system or --overwite-conf
    has to be specified. Since we need to manipulate it and can't guarantee order
     from Puppet / Ruby we just link to two on the primary-controller to avoid
     the issue.

    All additional users of config should config pull with --overwite-conf and
    not have this issue.

    The old link is left in place at this time to support older versions of
     ceph-deploy and can be removed in a later release.

    Closes-bug: #1333814
    Change-Id: I0b61c71effcf779801756968b8ed8b8cb93cb54b

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Reproduced on ISO #272
"build_id": "2014-06-26_00-31-14",
"mirantis": "yes",
"build_number": "272",
"ostf_sha": "4d2efa822344b6ca022ec4086b6f083c07d90e14",
"nailgun_sha": "83a057b0ab705955f763e59914aec513c9a08a7b",
"production": "docker",
"api": "1.0",
"fuelmain_sha": "86cd7a3cf72c0dc5d7f6c5ba4ad0ab6459681839",
"astute_sha": "694b5a55695e01e1c42185bfac9cc7a641a9bd48",
"release": "5.1",
"fuellib_sha": "337418f8d5aef8adf596e9fc1b95ceed7a3ba4a7"

1. Create new environment (CentOS, HA mode)
2. Choose VLAN segmentation
3. Choose both Ceph
4. Add 3 controllers, 2 computes, 2 ceph
5. Start deployment
6. There are errors on controllers:

2014-06-26 11:22:25 ERR

 (/Stage[main]/Ceph::Mon/Exec[ceph-deploy mon create]/returns) change from notrun to 0 failed: ceph-deploy mon create node-1:192.168.0.2 returned 1 instead of one of [0]

2014-06-26 11:49:23 ERR

 (/Stage[main]/Ceph::Conf/Exec[ceph-deploy gatherkeys remote]/returns) change from notrun to 0 failed: ceph-deploy gatherkeys node-1 returned 1 instead of one of [0]

Changed in fuel:
status: Fix Committed → Confirmed
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Controllers - node-1,2,3

Revision history for this message
Anastasia Palkina (apalkina) wrote :

Reproduced on ISO #281
"build_id": "2014-07-01_02-01-14",
"ostf_sha": "7171951202a22d8dfe9b3b76c9824001e2485630",
"build_number": "281",
"api": "1.0",
"nailgun_sha": "ff01812bb98e22acf136078368380b417b71d78c",
"production": "docker",
"fuelmain_sha": "dc2f49711946a34cfc94b479e87669eb614b4056",
"astute_sha": "6ed4d5a2d59675a59bb3f5bc94851ff13a4a4468",
"feature_groups": ["mirantis"],
"release": "5.1",
"fuellib_sha": "795737e3655623dbf932393b426659e8afcb77b9"

1. Create new environment (CentOS, HA mode)
2. Choose GRE segmentation
3. Choose Ceph for volumes
4. Add 3 controllers, compute, 2 ceph
5. Start deployment. It has failed
6. There are errors on first controller (node-7):

2014-07-01 13:30:40 ERR

 (/Stage[main]/Ceph::Mon/Exec[ceph-deploy mon create]/returns) change from notrun to 0 failed: ceph-deploy mon create node-7:192.168.0.7 returned 1 instead of one of [0]

Revision history for this message
Anastasia Palkina (apalkina) wrote :
Changed in fuel:
importance: High → Critical
Changed in fuel:
assignee: Andrew Woodward (xarses) → Dmitry Borodaenko (dborodaenko)
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Confirmed: logs in fuel-snapshot-2014-07-01_13-58-48.tgz still have the same "ceph-deploy mon create" failure as in the original report for this bug:

[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.2): /usr/bin/ceph-deploy mon create node-7:192.168.0.7
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts node-7:192.168.0.7
[ceph_deploy.mon][DEBUG ] detecting platform for host node-7 ...
[192.168.0.7][DEBUG ] connected to host: 192.168.0.7
[192.168.0.7][DEBUG ] detect platform information from remote host
[192.168.0.7][DEBUG ] detect machine type
[ceph_deploy.mon][INFO ] distro info: CentOS 6.5 Final
[node-7][DEBUG ] determining if provided host has same hostname in remote
[192.168.0.7][DEBUG ] get remote short hostname
[node-7][DEBUG ] deploying mon to node-7
[192.168.0.7][DEBUG ] get remote short hostname
[192.168.0.7][DEBUG ] remote hostname: node-7
[192.168.0.7][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.mon][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors

Changed in fuel:
status: Confirmed → Triaged
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

The reason the fix from https://review.openstack.org/102381 is insufficient is that the old problem with ceph.conf reordering is back in ceph-deploy 1.5.x.

In https://github.com/ceph/ceph-deploy/blob/v1.5.2/ceph_deploy/hosts/remotes.py#L68 write_conf() does a string comparison of the expected configuration with verbatim contents of target ceph.conf, where expected string matches the actual ceph.conf but is reordered.

Expected:

[global]
auth_service_required = cephx
osd_pool_default_size = 1
filestore_xattr_use_omap = true
osd_pool_default_min_size = 1
auth_client_required = cephx
osd_pool_default_pg_num = 256
auth_cluster_required = cephx
mon_host = 192.168.0.2
osd_journal_size = 2048
public_network = 192.168.0.0/24
mon_initial_members = node-1
osd_mkfs_type = xfs
auth_supported = cephx
cluster_network = 192.168.1.0/24
fsid = ea1533ec-4eaa-498d-a6b3-b10236bfd21b

Actual:

[global]
auth_service_required = cephx
filestore_xattr_use_omap = true
auth_client_required = cephx
auth_cluster_required = cephx
mon_host = 192.168.0.2
mon_initial_members = node-1
fsid = ea1533ec-4eaa-498d-a6b3-b10236bfd21b
osd_pool_default_size = 1
osd_pool_default_min_size = 1
osd_pool_default_pg_num = 256
public_network = 192.168.0.0/24
osd_journal_size = 2048
auth_supported = cephx
osd_mkfs_type = xfs
cluster_network = 192.168.1.0/24

This only happens on CentOS 6.x (might be due to Python 2.6), on Ubuntu "ceph-deploy mon create" completes successfully.

Changed in fuel:
importance: Critical → High
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :
Changed in fuel:
status: Triaged → In Progress
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :
Changed in fuel:
assignee: Dmitry Borodaenko (dborodaenko) → Alexander Podrepniy (apodrepniy)
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Please prepare a ceph-deploy package based on the latest upstream release (1.5.7) plus the attached patch. Beware that this patch introduces dependency on python-ordereddict on CentOS. When testing, please make sure you start with an ISO image that includes the updated ceph-deploy RPM/DEB, so that the package is in place in the repositories on Fuel master _before_ deployment.

Changed in fuel:
assignee: Alexander Podrepniy (apodrepniy) → MOS Linux (mos-linux)
Changed in fuel:
assignee: MOS Linux (mos-linux) → Kostiantyn Danylov (kdanylov)
no longer affects: fuel/5.0.x
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Upped priority to Critical since this now blocks testing of 5.1.

Changed in fuel:
importance: High → Critical
Revision history for this message
Kostiantyn Danylov (kdanylov) wrote :

I'm testing fix at the moment (proposed ordereddict patch 1.5.9 ceph-deploy)

Revision history for this message
Kostiantyn Danylov (kdanylov) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceph-deploy has been built from changeset: http://gerrit.mirantis.com/18829
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-18829/ubuntu
You can build an ISO with this package:
make iso EXTRA_DEB_REPOS="http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-18829/ubuntu /"

Revision history for this message
Kostiantyn Danylov (kdanylov) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceph-deploy has been built from changeset: http://gerrit.mirantis.com/18831
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-18831/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-18831/centos"

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceph-deploy has been built from changeset: http://gerrit.mirantis.com/18828
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-18828/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable-18828/centos"

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceph-deploy has been built from changeset: http://gerrit.mirantis.com/18831
RPM Repository URL: http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos
You can build an ISO with this package:
make iso EXTRA_RPM_REPOS="osci-testing,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.1-stable/centos"

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceph-deploy has been built from changeset: http://gerrit.mirantis.com/18829
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-18829/ubuntu
You can build an ISO with this package:
make iso EXTRA_DEB_REPOS="http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-18829/ubuntu /"

Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceph-deploy has been built from changeset: http://gerrit.mirantis.com/18829
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable/ubuntu
You can build an ISO with this package:
make iso EXTRA_DEB_REPOS="http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable/ubuntu /"

Changed in fuel:
status: In Progress → Fix Released
status: Fix Released → Fix Committed
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Download full text (4.8 KiB)

Issue was reproduced in latest system tests
{

    "build_id": "2014-07-28_02-01-14",
    "ostf_sha": "8c328521b1444f22c50463b9432193e20ed33813",
    "build_number": "366",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "83cc9ed44ebc8dd97248483b6d414ebbc4cff3c0",
    "production": "docker",
    "fuelmain_sha": "9adfbf5a52cedbdd16ec1a74f6c44c5b3419b87c",
    "astute_sha": "aa5aed61035a8dc4035ab1619a8bb540a7430a95",
    "feature_groups": [
        "mirantis"
    ],
    "release": "5.1",
    "fuellib_sha": "d1c7f67b3cf51978d3178c8666ea398f2477dcb5"

}
Steps:
1. Cluster - Ubuntu, Simple, Flat Nova-network, Ceph for volumes, Ceph for images, Ceph RadosGW for objects, 1 controller, 1 compute, 3 ceph nodes
2. Deploy cluster
Expected - deployment passed
Actual - (/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd prepare]/returns) change from notrun to 0 failed: ceph-deploy osd prepare node-2:/dev/vdb2 node-2:/dev/vdc2 returned 1 instead of one of [0]

2014-07-28 09:58:03 ERR

 /usr/bin/puppet:4

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:91:in `execute'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util.rb:478:in `exit_on_fail'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application.rb:470:in `plugin_hook'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application.rb:364:in `run'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:146:in `run_command'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:218:in `main'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:268:in `apply_catalog'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `run'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:124:in `apply_catalog'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `benchmark'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:125:in `apply_catalog'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:163:in `apply'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction/report.rb:108:in `as_logging_destination'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/util/log.rb:149:in `with_destination'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:164:in `apply'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/transaction.rb:108:in `evaluate'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:118:in `traverse'

2014-07-28 09:58:03 ERR

 /usr/lib/ruby...

Read more...

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Changed in fuel:
status: Fix Committed → Confirmed
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

The failure from comment #24 was due to bug #1333814. Please check existing bugs before re-opening old bugs for unrelated failures.

Changed in fuel:
status: Confirmed → Fix Committed
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Actually this is still a problem on Ubuntu: the deb packages above do not include my patch. Please fix and re-upload ASAP.

Changed in fuel:
status: Fix Committed → Triaged
Revision history for this message
Kostiantyn Danylov (kdanylov) wrote :

Will commit fix shortly

Revision history for this message
Kostiantyn Danylov (kdanylov) wrote :
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceph-deploy has been built from changeset: http://gerrit.mirantis.com/19635
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-19635/ubuntu
You can build an ISO with this package:
make iso EXTRA_DEB_REPOS="http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable-19635/ubuntu /"

Changed in fuel:
status: Triaged → In Progress
Revision history for this message
OSCI Robot (oscirobot) wrote :

Package ceph-deploy has been built from changeset: http://gerrit.mirantis.com/19635
DEB Repository URL: http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable/ubuntu
You can build an ISO with this package:
make iso EXTRA_DEB_REPOS="http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.1-stable/ubuntu /"

Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

This ceph-deploy package in 5.1 package repo for ubuntu includes the patch:
http://fuel-repository.mirantis.com/fwm/5.1/ubuntu/pool/main/ceph-deploy_1.5.9-ubuntu5_all.deb

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Reproduced on master ISO #378
"build_id": "2014-07-31_10-30-25",
"ostf_sha": "b4c5efa51909404fd9ec1d0bbc38a31b200e1d6d",
"build_number": "378",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "6119e1017771a704d0ff6dc84299801d211748b2",
"production": "docker",
"fuelmain_sha": "63d0775708b0f5fa4d6d1e09a316d9c26f7e5444",
"astute_sha": "b16efcec6b4af1fb8669055c053fbabe188afa67",
"feature_groups": ["mirantis", "experimental"],
"release": "5.1",
"fuellib_sha": "9a64237c70fc464ef0b11ac7c0bad34e8c202135"

1. Create new environment (Ubuntu, HA mode)
2. Choose VLAN segmentation
3. Choose both Ceph
4. Add 3 controllers+ceph, compute
5. Start deployment. It has failed.
6. There are errors on controller (node-6):

2014-07-31 12:29:44 ERR

 (/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd activate]/returns) change from notrun to 0 failed: ceph-deploy osd activate node-6:/dev/sdb2 node-6:/dev/sdc2 returned 1 instead of one of [0]

7. There are errors on compute node (node-9):

2014-07-31 12:29:17 ERR

 (/Stage[main]/Ceph::Conf/Exec[ceph-deploy config pull]/returns) change from notrun to 0 failed: ceph-deploy --overwrite-conf config pull node-6 returned 1 instead of one of [0]

2014-07-31 12:35:23 ERR

 (/Stage[main]/Ceph::Conf/Exec[ceph-deploy gatherkeys remote]/returns) change from notrun to 0 failed: ceph-deploy gatherkeys node-6 returned 1 instead of one of [0]

Changed in fuel:
status: Fix Committed → Confirmed
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Controllers -node-6,7,8
Compute - node-9

Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

The failure in commit #33 is due to an unrelated problem:

2014-07-31T12:29:44.193624+01:00 notice: (/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd activate]/returns) [node-6][INFO ] Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdc2
2014-07-31T12:29:44.193624+01:00 notice: (/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd activate]/returns) [node-6][DEBUG ] === osd.3 ===
2014-07-31T12:29:44.199063+01:00 notice: (/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd activate]/returns) [node-6][DEBUG ] failed: 'timeout 30 /usr/bin/ceph -c /etc/ceph/ceph.conf --name=osd.3 --keyri
ng=/var/lib/ceph/osd/ceph-3/keyring osd crush create-or-move -- 3 0.06 host=node-6 root=default'
2014-07-31T12:29:44.208757+01:00 notice: (/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd activate]/returns) [node-6][WARNING] INFO:ceph-disk:ceph osd.3 already mounted in position; unmounting ours.
2014-07-31T12:29:44.208757+01:00 notice: (/Stage[main]/Ceph::Osd/Exec[ceph-deploy osd activate]/returns) [node-6][WARNING] ceph-disk: Error: ceph osd start failed: Command '['/usr/sbin/service', 'ceph', 'start', 'osd.3']' returned non-zero exit status 1

Soon after that, node-6 (primary controller) went offline, and deployment of the rest of the cluster failed. The failure above by itself could be related to bug #1322230, the controller going offline to bug #1348839, or this combination of symptoms could be indicative of a new bug.

Bug description updated to help identify reoccurences of this problem and tell them from all other ceph failures.

description: updated
Changed in fuel:
status: Confirmed → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/114475

Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

This bug was caused by ceph-deploy upgrade to 1.5.x in 5.1, it wasn't present in 5.0.x where ceph-deploy is still 1.2.x.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/5.0)

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/114475

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package ceph-deploy has been built for project packages/trusty/ceph-deploy
Package version == 1.5.20, package release == ubuntu1

Changeset: https://review.fuel-infra.org/3035
project: packages/trusty/ceph-deploy
branch: 6.1
author: Alexei Sheplyakov
committer: Alexei Sheplyakov
subject: Added ceph-deploy version 1.5.20 for Ubuntu 14.04
status: patchset-created

Files placed on repository:
ceph-deploy_1.5.20-0ubuntu1~mos6.1+1_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://osci-obs.vm.mirantis.net:82/trusty-fuel-6.1-stable-3035/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package ceph-deploy has been built for project packages/trusty/ceph-deploy
Package version == 1.5.20, package release == ubuntu1

Changeset: https://review.fuel-infra.org/3035
project: packages/trusty/ceph-deploy
branch: 6.1
author: Alexei Sheplyakov
committer: Alexei Sheplyakov
subject: Added ceph-deploy version 1.5.20 for Ubuntu 14.04
status: patchset-created

Files placed on repository:
ceph-deploy_1.5.20-0ubuntu1~mos6.1+1_all.deb

NOTE: Changeset is not merged, created temporary package repository.
DEB repository URL: http://obs-1.mirantis.com:82/trusty-fuel-6.1-stable-3035/ubuntu

Revision history for this message
OSCI Robot (oscirobot) wrote :

DEB package ceph-deploy has been built for project packages/trusty/ceph-deploy
Package version == 1.5.20, package release == ubuntu1

Changeset: https://review.fuel-infra.org/3035
project: packages/trusty/ceph-deploy
branch: 6.1
author: Alexei Sheplyakov
committer: Alexei Sheplyakov
subject: Added ceph-deploy version 1.5.20 for Ubuntu 14.04
status: change-merged

Files placed on repository:
ceph-deploy_1.5.20-0ubuntu1~mos6.1+1_all.deb

Changeset merged. Package placed on primary repository
DEB repository URL: http://obs-1.mirantis.com:82/trusty-fuel-6.1-stable/ubuntu

Changed in fuel:
status: Fix Committed → Fix Released
Changed in fuel:
status: Fix Released → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.