libvirt.yaml failed to load

[raghunath]

Followed the below link to install monasa using devstack.
https://github.com/openstack/monasca-api/tree/master/devstack

For vm stats copied libvirt.yam from

/opt/stack/monasca-agent/conf.d/libvirt.yaml.example to /etc/monasca/agent/conf.d/libvirt.yaml and restartednasca-agent.
Restart is succeful but the libvirt plugin failed to load with the below error (in collector.log):

2016-03-20 13:44:47 EDT | WARNING | collector | monasca_agent.collector.checks.check.libvirt(libvirt.py:228) | Metrics cache missing or corrupt, rebuilding.
2016-03-20 13:44:47 EDT | ERROR | collector | monasca_agent.collector.checks.check.libvirt(check.py:565) | Check 'libvirt' instance #0 failed
Traceback (most recent call last):
  File "/opt/monasca-agent/local/lib/python2.7/site-packages/monasca_agent/collector/checks/check.py", line 556, in run
    self.check(instance)
  File "/opt/monasca-agent/lib/python2.7/site-packages/monasca_agent/common/../collector/checks_d/libvirt.py", line 425, in check
    for inst in insp._get_connection().listAllDomains():
AttributeError: 'Inspector' object has no attribute '_get_connection'
2016-03-20 13:45:02 EDT | WARNING | collector | monasca_agent.collector.checks.check.libvirt(libvirt.py:228) | Metrics cache missing or corrupt, rebuilding.
2016-03-20 13:45:02 EDT | ERROR | collector | monasca_agent.collector.checks.check.libvirt(check.py:565) | Check 'libvirt' instance #0 failed
Traceback (most recent call last):
  File "/opt/monasca-agent/local/lib/python2.7/site-packages/monasca_agent/collector/checks/check.py", line 556, in run
    self.check(instance)
  File "/opt/monasca-agent/lib/python2.7/site-packages/monasca_agent/common/../collector/checks_d/libvirt.py", line 425, in check
    for inst in insp._get_connection().listAllDomains():
AttributeError: 'Inspector' object has no attribute '_get_connection'

[Laszlo Hegedus (laszlo-hegedus)]

Hi,

I managed to reproduce the issue in DevStack on current master.

2016-05-03 08:01:04 CEST | ERROR | collector | monasca_agent.collector.checks.check.libvirt(check.py:248) | Check 'libvirt' instance #0 failed
Traceback (most recent call last):
  File "/opt/monasca-agent/local/lib/python2.7/site-packages/monasca_agent/collector/checks/check.py", line 246, in run
    self.check(instance)
  File "/opt/monasca-agent/lib/python2.7/site-packages/monasca_agent/common/../collector/checks_d/libvirt.py", line 428, in check
    for inst in insp._get_connection().listAllDomains():
AttributeError: 'Inspector' object has no attribute '_get_connection'

I'd like to look into this and try to find a solution if no one is working on it right now.

[raghunath (raghunath-d)]

Hi,

 The issue related to 1559758 is resolved by installing libvirt-python,lxml,nova and neutron clients.

/opt/monasca-agent/bin/pip install libvirt-python
/opt/monasca-agent/bin/pip install lxml

But still monasca is unable to monitor the newly created vm.

Here is the log:
2016-04-19 00:13:28 MDT | ERROR | collector | monasca_agent.collector.checks.check.libvirt(libvirt.py:451) | instance-00000002 is not known to nova after instance cache update -- skipping this ghost VM.
Already bug is raised on the above issue and it is in closed state,but it is not working in the latest setup.
https://bugs.launchpad.net/monasca/+bug/1486688

[Laszlo Hegedus (laszlo-hegedus)]

Hi,

thanks. I got to the same conclusion with the missing packages. :)

But I don't get the other error you mentined. Monitoring of VMs seems to work fine now. I installed my environment yesterday. Maybe some recent change on master has fixed it.

[raghunath (raghunath-d)]

Hi,

 The issue related to 1559758 is resolved by installing libvirt-python
,lxml,nova and neutron clients.

/opt/monasca-agent/bin/pip install libvirt-python
/opt/monasca-agent/bin/pip install lxml

But still monasca is unable to monitor the newly created vm.

Here is the log:
2016-04-19 00:13:28 MDT | ERROR | collector |
monasca_agent.collector.checks.check.libvirt(libvirt.py:451) |
instance-00000002 is not known to nova after instance cache update --
skipping this ghost VM.
Already bug is raised on the above issue and it is in closed state,but it
is not working in the lates setup.
https://bugs.launchpad.net/monasca/+bug/1486688

With Best Regards
Raghunath Dudyala
Tata Consultancy Services Limited
Mailto: <email address hidden>
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
                        Business Solutions
                        Consulting
____________________________________________

From: Laszlo Hegedus <email address hidden>
To: <email address hidden>
Date: 05/03/2016 12:05 PM
Subject: [Bug 1559758] Re: libvirt.yaml failed to load
Sent by: <email address hidden>

Hi,

I managed to reproduce the issue in DevStack on current master.

2016-05-03 08:01:04 CEST | ERROR | collector |
monasca_agent.collector.checks.check.libvirt(check.py:248) | Check
'libvirt' instance #0 failed
Traceback (most recent call last):
  File
"/opt/monasca-agent/local/lib/python2.7/site-packages/monasca_agent/collector/checks/check.py",
line 246, in run
    self.check(instance)
  File
"/opt/monasca-agent/lib/python2.7/site-packages/monasca_agent/common/../collector/checks_d/libvirt.py",
line 428, in check
    for inst in insp._get_connection().listAllDomains():
AttributeError: 'Inspector' object has no attribute '_get_connection'

I'd like to look into this and try to find a solution if no one is
working on it right now.

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1559758

Title:
  libvirt.yaml failed to load

Status in Monasca:
  New

Bug description:
  Followed the below link to install monasa using devstack.
  https://github.com/openstack/monasca-api/tree/master/devstack

  For vm stats copied libvirt.yam from

  /opt/stack/monasca-agent/conf.d/libvirt.yaml.example to
/etc/monasca/agent/conf.d/libvirt.yaml and restartednasca-agent.
  Restart is succeful but the libvirt plugin failed to load with the below
error (in collector.log):

  2016-03-20 13:44:47 EDT | WARNING | collector |
monasca_agent.collector.checks.check.libvirt(libvirt.py:228) | Metrics
cache missing or corrupt, rebuilding.
  2016-03-20 13:44:47 EDT | ERROR | collector |
monasca_agent.collector.checks.check.libvirt(check.py:565) | Check
'libvirt' instance #0 failed
  Traceback (most recent call last):
    File
"/opt/monasca-agent/local/lib/python2.7/site-packages/monasca_agent/collector/checks/check.py",
line 556, in run
      self.check(instance)
    File
"/opt/monasca-agent/lib/python2.7/site-packages/monasca_agent/common/../collector/checks_d/libvirt.py",
line 425, in check
      for inst in insp._get_conn...

[raghunath (raghunath-d)]

Hi,

Followed the below link for installing monasca:

https://github.com/openstack/monasca-api/tree/master/devstack

Login dashboard using username:mini-mon,password:password.

Launched one vm from the dashboard.
Still I am able to see same error:

2016-05-03 03:48:58 MDT | ERROR | collector | monasca_agent.collector.checks.check.libvirt(libvirt.py:454) | instance-00000001 is not known to nova after instance cache update -- skipping this ghost VM.
2016-05-03 03:49:13 MDT | ERROR | collector | monasca_agent.collector.checks.check.libvirt(libvirt.py:454) | instance-00000001 is not known to nova after instance cache update -- skipping this ghost VM.

Please let me know do I missing something ?

[Laszlo Hegedus (laszlo-hegedus)]

Unfortunately I'm quite new to Monasca. Here's my libvirt.yaml, maybe I set it up differently:

init_config:
    admin_password: <pw>
    admin_tenant_name: admin
    admin_user: admin
    identity_uri: 'http://127.0.0.1:35357/v2.0'
    cache_dir: /dev/shm
    nova_refresh: 60
    vm_probation: 60
    ping_check: false
    alive_only: false
    metadata:
    - scale_group
instances:
    - {}

I remember changing nova_refresh, vm_probation and ping_check from the defaults.

[Laszlo Hegedus (laszlo-hegedus)]

Raghunath,

do you still have this issue or did you manage to fix it?

[raghunath (raghunath-d)]

Hi Laszlo,

I have similar libvirt.yaml as mentione in your previous comment.

Still I have the below issue:
check.libvirt(libvirt.py:454) | instance-00000001 is not known to nova after instance cache update -- skipping this ghost VM.

[Laszlo Hegedus (laszlo-hegedus)]

I'll try to reproduce the issue and figure out what goes wrong. I would guess that the instance was deleted from Nova database, but it was not removed from the host.

[Laszlo Hegedus (laszlo-hegedus)]

Well, after deleting all the related rows from nova database, I got the error message. I used the libvirt check from latest master.

2016-06-27 04:13:53 EDT | ERROR | collector | monasca_agent.collector.checks.check.libvirt(libvirt.py:610) | instance-00000014 is not known to nova after instance cache update -- skipping this ghost VM.

But this is quite an extreme case. Are you sure that you don't have any "garbage" left in libvirt on your compute node?

I think the check works as intended.

[Ryan (ryan-brandt)]

Any further information on this error? Has this been resolved?

[Bob Lane (anustsn)]

I faced same issue. Is there anyone who solved this issue?

[Konstantinos Mouzakitis (mouza8)]

I'm facing the same issue. Just to clarify, this is for existing/running VMs and not for deleted ones. Let us know if there's a way to fix this. Thanks!

[Mariusz Karpiarz (mkarpiarz)]

Anyone seeing error messages mentioning "ghost VMs" please check logs for warnings like this one:

```
2020-09-03 09:17:16 UTC | WARNING | collector | monasca_agent.collector.checks.check.libvirt(libvirt.py:160) | No 'nova-compute' service found on host: <hostname>
```
Then make sure that the "hostname" parameter for your Agent Collector matches the hypervisor's name on the list from `openstack compute service list`.

See: https://bugs.launchpad.net/kolla-ansible/+bug/1894076