Ceph doesn't work correctly with Calamari script

[Andrey Grebennikov]

Fuel 6.1, Ubuntu

We have Ceph Firefly 0.80.9 installed with Fuel.

Customer tries to install Calamari for better monitoring of Ceph environment.

Calamari sets up an option in ceph.conf:
osd crush location hook = /usr/bin/calamari-crush-location

When Calamari/User tries to execute the script it returns an error like:
# calamari-crush-location --cluster ceph --id 200 --type osd
ERROR:calamari_osd_location:Error 13 running ceph config-key get:'Error EACCES: access denied'
ERROR:calamari_osd_location:Error 2 running ceph config-key get:'Error ENOENT: error obtaining 'daemon-private/osd.200/v1/calamari/osd_crush_location': (2) No such file or directory'
host=node-10

This is potentially connected with the upstream bug:
http://tracker.ceph.com/issues/10844

and resolved in the next minor version of Firefly 0.80.10.

We need to build 0.80.10 with our changes and allow the existing environments to update the version of Ceph, as well as create an instruction with correct procedure of update.

[Vitaly Sedelnik (vsedelnik)]

Invalid for 7.0x and 8.0.x as MOS 7.0 comes with Ceph 0.80.9 Firefly

[Vitaly Sedelnik (vsedelnik)]

Sorry, misread - as MOS 7.0 comes with Ceph 0.80.9 Firefly it's Confirmed for both 7.0 and 8.0

[Herman Narkaytis (hnarkaytis)]

Is it a bug for MU or for 8.0 GA?

In 8.0 we expect to have 0.9.4 Hammer. This should resolve the issue.
For MU we can apply patch and provide custom build.

[Polina Petriuk (ppetriuk)]

Firefly 0.80.10 has a lot of bug-fixes [1], with some related to memory leaks, etc. So I think we should consider including it as part of the 6.1 and 7.0 versions. However if that is not possible, would be great to at least include the fix for this current issue into 6.1 and 7.0 MUs.

[1] - http://docs.ceph.com/docs/master/release-notes/#v0-80-10-firefly

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 7.0
Change author: Alexei Sheplyakov <email address hidden>
Review: https://review.fuel-infra.org/13509

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 8.0
Change author: Alexei Sheplyakov <email address hidden>
Review: https://review.fuel-infra.org/13608

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexei Sheplyakov <email address hidden> on branch: 7.0
Review: https://review.fuel-infra.org/13509
Reason: The correct way is to update the code in 8.0 branch and cherry-pick the patch to 7.0 (and possibly 6.1) branches

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: 8.0
Change author: Alexei Sheplyakov <email address hidden>
Review: https://review.fuel-infra.org/13879

[Denis Meltsaykin (dmeltsaykin)]

Setting this as Won't fix for 6.1-updates and 7.0-updates, as such a big update is not possible in the scope of maintenance updates.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/13879
Submitter: Pkgs Jenkins <email address hidden>
Branch: 8.0

Commit: cc0c175908eb0d44c6403f3eedf64a96af86c1dc
Author: Alexei Sheplyakov <email address hidden>
Date: Thu Nov 12 11:33:30 2015

Update to ceph 0.94.5

... which is the latest Hammer release at the moment.
The source has been downloaded from

http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/ceph_0.94.5.orig.tar.bz2
http://archive.ubuntu.com/ubuntu/pool/main/c/ceph/ceph_0.94.5-0ubuntu1.debian.tar.xz

Related-Bug: #1509498
Change-Id: Idc8be5cc6d95b3072361e5ffc3a004087e52fe5d

[Alexei Sheplyakov (asheplyakov)]

Ceph 0.94.5 which has been merged [1] to MOS 8.0 contains a fix [2].

[1] https://review.fuel-infra.org/13879
[2] https://github.com/ceph/ceph/pull/3710

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Alexei Sheplyakov <email address hidden> on branch: 8.0
Review: https://review.fuel-infra.org/13608
Reason: Hammer (0.94.5) will be shipped with MOS 8.0

[Maksym Shalamov (mshalamov)]

The issue reproduced on MOS 8.0 with ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43).

root@node-4:~# calamari-crush-location --cluster ceph --id 4 --type osd
ERROR:calamari_osd_location:Error 2 running ceph config-key get:'Error ENOENT: error obtaining 'daemon-private/osd.4/v1/calamari/osd_crush_location': (2) No such file or directory'
ERROR:calamari_osd_location:Error 2 running ceph config-key get:'Error ENOENT: error obtaining 'daemon-private/osd.4/v1/calamari/osd_crush_location': (2) No such file or directory'
host=node-4

[Maksym Shalamov (mshalamov)]

Calamary version:
Calamari-Server: 1.3.3-2-gd363ecf
Calamari-Client: v1.2.2-32-931ee58-dirty

[Maksym Shalamov (mshalamov)]

Verified on:

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "589"
  build_id: "589"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "643a1ef27c7dccc1c2a2ad26b85c09226b35a67d"