Ceilometer: Disk.usage metric is not available for ceph volume.

[Alexey]

it's MOS 7.0 issue.

[STEPS]
1. Create VM instance with ceph volume attached.

2. Checked disk.usage metric data from Mongodb.

[RESULT]
- No disk.usage metric. The computing node where the VM is launched show error messag:
  ERROR ceilometer.compute.pollsters.disk [-] Ignoring instance instance-00001401 (d26e8953-672c-47dd-817d-fa3cd753361e) : internal error: missing storage backend for network files using rbd protocol

Compute node: /var/log/ceilometer/ceilometer-agent-compute.log

2016-01-27 01:25:28.744 11511 ERROR ceilometer.compute.pollsters.disk [-] Ignoring instance instance-00001401 (d26e8953-672c-47dd-817d-fa3cd753361e) : internal error: missing storage backend for network files using rbd protocol
2016-01-27 01:25:28.744 11511 TRACE ceilometer.compute.pollsters.disk Traceback (most recent call last):
2016-01-27 01:25:28.744 11511 TRACE ceilometer.compute.pollsters.disk File "/usr/lib/python2.7/dist-packages/ceilometer/compute/pollsters/disk.py", line 693, in get_samples
2016-01-27 01:25:28.744 11511 TRACE ceilometer.compute.pollsters.disk instance,
2016-01-27 01:25:28.744 11511 TRACE ceilometer.compute.pollsters.disk File "/usr/lib/python2.7/dist-packages/ceilometer/compute/pollsters/disk.py", line 662, in _populate_cache
2016-01-27 01:25:28.744 11511 TRACE ceilometer.compute.pollsters.disk for disk, info in disk_info:
2016-01-27 01:25:28.744 11511 TRACE ceilometer.compute.pollsters.disk File "/usr/lib/python2.7/dist-packages/ceilometer/compute/virt/libvirt/inspector.py", line 209, in inspect_disk_info
2016-01-27 01:25:28.744 11511 TRACE ceilometer.compute.pollsters.disk block_info = domain.blockInfo(device)
2016-01-27 01:25:28.744 11511 TRACE ceilometer.compute.pollsters.disk File "/usr/lib/python2.7/dist-packages/libvirt.py", line 654, in blockInfo
2016-01-27 01:25:28.744 11511 TRACE ceilometer.compute.pollsters.disk if ret is None: raise libvirtError ('virDomainGetBlockInfo() failed', dom=self)
2016-01-27 01:25:28.744 11511 TRACE ceilometer.compute.pollsters.disk libvirtError: internal error: missing storage backend for network files using rbd protocol

seems known issue: https://bugs.launchpad.net/ceilometer/+bug/1457440

It's important for the customer, because they need to provide a monitoring&alarm service for customer's VM.

[Roman Podoliaka (rpodolyaka)]

According to comments in https://bugs.launchpad.net/ceilometer/+bug/1457440 this seems to be a libvirt issue

[Roman Podoliaka (rpodolyaka)]

I think this must not be a blocker for us in 8.0, thus setting the importance to Medium and closing as Won't Fix.

[Roman Podoliaka (rpodolyaka)]

This will probably require a RCA and upgrade of libvirt, please escalate, if you want to see this fixed in -updates.

[Timur Nurlygayanov (tnurlygayanov)]

Hi maintanence team, could you please check if it is possible to include the fix for MOS 7.0 and MOS 8.0 updates?

[Ivan Suzdal (isuzdal)]

Could you provide access to environment for me? I need it for debugging purposes.

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Javier Diaz Jr (javierdiazcharles)]

I have a Mirantis customer experiencing this issue. There isn't too much of a concern in terms of functionality, but the immediate concern is that the error messages are increasing the size of the logs as the error message repeats constantly. Do we have a workaround to suppress the error message or other way to avoid this in the mean time?

[Dmitry Teselkin (teselkin-d)]

@Javier, we don't have a workaround right now, since we can't reproduce the issue.
Is it possible to get access to the env for debugging?
BTW, what MOS release is deployed there?

[Javier Diaz Jr (javierdiazcharles)]

We can possibly do a WebEx with the customer if thats something you'd be interested in. They are running MOS 7.0.

My colleague was also able to reproduce this in his MOS 7.0 MU2 lab yesterday:

2016-03-29 02:03:34.692+0000: 6023: error : virStorageFileBackendForTypeInternal:1232 : internal error: missing storage backend for network files using rbd protocol

Perhaps that might be accessible, but I'd have to reach out to him. Let me know how you would like to proceed.

[Javier Diaz Jr (javierdiazcharles)]

Hey Team, we can do a WebEx or Hangouts with our lab where we are seeing the problem. Just let us know when. We are available from 12PM CDT - 8PM CDT.

[Maxim V. Yefimov (mefimov)]

Here is customer's current patch to disable to call the problematic function, blockInfo, which generates a huge number of error message unnecessarily.

    def inspect_disk_info(self, instance):

        domain = self._get_domain_not_shut_off_or_raise(instance)

        tree = etree.fromstring(domain.XMLDesc(0))

#JunPark: April 4, 16: to avoid false errors due to a bug of blockInfo with Ceph

        for device in filter(

                bool,

                [target.get("dev")

                 for target in tree.findall('devices/disk/target')]):

            disk = virt_inspector.Disk(device=device)

# block_info = domain.blockInfo(device)

# info = virt_inspector.DiskInfo(capacity=block_info[0],

# allocation=block_info[1],

# physical=block_info[2])

            info = virt_inspector.DiskInfo(capacity=0,

                                           allocation=0,

                                           physical=0)

            yield (disk, info)

[Dmitry Teselkin (teselkin-d)]

Sorry for a delay with an answer.
We were able to reproduce the issue in our lab so we investigated it, here is what we’ve found:

* First of all, libvirt doesn’t support RBD pools for backing volume disks out of the box [1]. There is a patch that adds it but it wasn’t merged [2] so it’s not an option here.

* CEPH volumes attached to libvirt according to CEPH official documentation [3], please note that it requires type=‘network’ in ‘disk definition.

* Actual failure occurs in libvirt in ‘virStorageFileBackendForTypeInternal’ function [4] because there is no item in ‘fileBackends’ that is of type ‘network’ and protocol ‘rbd’. Looking at definition of ‘fileBackends’ [5] it becomes clear that there is no support for that case too, since the only backend with type ‘network’ is ‘virStorageFileBackendGluster’.

So there is no way to gather disk usage data if RBD is used as backend.

[1] https://www.redhat.com/archives/libvirt-users/2015-April/msg00001.html
[2] https://www.redhat.com/archives/libvir-list/2014-March/msg00107.html
[3] http://docs.ceph.com/docs/hammer/rbd/libvirt/#configuring-the-vm
[4] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/storage/storage_backend.c;h=70d2a59cc857db1d4147e3d976374cb6b7ed37eb;hb=09661a8e2910d4f71ff0d55d0e86f8699c369bb9#l1209
[5] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/storage/storage_backend.c;h=70d2a59cc857db1d4147e3d976374cb6b7ed37eb;hb=09661a8e2910d4f71ff0d55d0e86f8699c369bb9#l138

[Javier Diaz Jr (javierdiazcharles)]

Understood, considering this, there is still the issue of the error being logged in the logs files (logs grow to large). If there is no solution I would suggest providing a way to suppress the error message. The customer simply commented out the the section that generates the error, perhaps removing the code would do the job. Alternatively, and I think this is a better way, a `try-except` would be a better way to go around this.

[Dmitry Teselkin (teselkin-d)]

I'm reassigning it to ceilometer team since they should know better.

[Dina Belova (dbelova)]

Added move-to-10.0 tag due to the fact bug was transferred from 9.0 to 10.0

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Ildar Svetlov <email address hidden>
Review: https://review.fuel-infra.org/23351

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Ildar Svetlov <email address hidden>
Review: https://review.fuel-infra.org/23361

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Ildar Svetlov <email address hidden>
Review: https://review.fuel-infra.org/23409

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ildar Svetlov <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/23351

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Ildar Svetlov <email address hidden>
Review: https://review.fuel-infra.org/23415

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ildar Svetlov <email address hidden> on branch: openstack-ci/fuel-7.0/2015.1.0
Review: https://review.fuel-infra.org/23361

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0/mitaka
Change author: Ildar Svetlov <email address hidden>
Review: https://review.fuel-infra.org/23418

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ildar Svetlov <email address hidden> on branch: 9.0/mitaka
Review: https://review.fuel-infra.org/23350

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0/mitaka
Change author: Ildar Svetlov <email address hidden>
Review: https://review.fuel-infra.org/23460

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ildar Svetlov <email address hidden> on branch: 9.0/mitaka
Review: https://review.fuel-infra.org/23418

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/23460
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: a216e536fdbdf5db2436a7cb3a30fe5d70bdab7b
Author: Ildar Svetlov <email address hidden>
Date: Wed Jul 20 13:55:23 2016

Retrieval of RBD device information issue

Libvirt still doesnt support retrieval of RBD device information.
Fix skip "network" type devices and send message to LogWarning.

Change-Id: Idf8949ed5d025bd249f239c597078f48a52f272d
Partial-Bug: 1539253

[Alexey Stupnikov (astupnikov)]

Steps to reproduce:
1. Deploy environment with 3 controllers/mongo and 3 compute/ceph nodes. Ceph should be used as a backend for all cloud storages.
2. Create VM with volume from image. Wait for 10 minutes to get usage metrics collected.
3. Search compute logs for errors:
# grep -r 'internal error: missing storage backend for network files using rbd protocol' /var/log/ceilometer-all.log | wc -l

[Alexander Gubanov (ogubanov)]

Verified on MOS 9.0 (RC2 snapshot 315) - fixed!
Proof/details: https://paste.mirantis.net/show/2698/
Moved to "Fix Release"

[Vladimir Jigulin (vjigulin)]

Verified on mos7.0+mu6-proposed

root@node-11:~# grep "Inspection disk usage of network disk" /var/log/ceilometer-all.log
<132>Oct 6 13:18:12 node-11 ceilometer-agent-compute Inspection disk usage of network disk 0871dda7-24a7-464d-b5ff-126f6a41c80c unsupported by libvirt
<132>Oct 6 13:19:12 node-11 ceilometer-agent-compute Inspection disk usage of network disk 0871dda7-24a7-464d-b5ff-126f6a41c80c unsupported by libvirt

[Ekaterina Shutova (eshutova)]

Verified on MOS 8.0 + MU4 updates.

Before there was error messages on compute nodes:
10.109.15.6
<131>Feb 13 13:07:41 node-4 ceilometer-polling: 2017-02-13 13:07:41.609 29729 ERROR ceilometer.compute.pollsters.disk [-] Ignoring instance instance-00000001 (ea136894-41d8-486a-a8ed-dd227586a601) : internal error: missing storage backend for network files using rbd protocol
2017-02-13 13:07:41.609 29729 ERROR ceilometer.compute.pollsters.disk libvirtError: internal error: missing storage backend for network files using rbd protocol
With updates log doesn't contain errors.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/newton
Change author: Ildar Svetlov <email address hidden>
Review: https://review.fuel-infra.org/33392

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/ocata
Change author: Ildar Svetlov <email address hidden>
Review: https://review.fuel-infra.org/34511

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ilya Tyaptin <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/33392

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Ilya Tyaptin <email address hidden> on branch: mcp/ocata
Review: https://review.fuel-infra.org/34511

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Roman Podoliaka <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/33809
Reason: we don't use 11.0/ocata anymore - mcp/ocata is the correct branch name