| 2021-06-01 16:38:20 |
Zakhar Kirpichenko |
bug |
|
|
added bug |
| 2021-06-01 16:38:20 |
Zakhar Kirpichenko |
attachment added |
|
Ceilometer and Nova configuration https://bugs.launchpad.net/bugs/1930446/+attachment/5501709/+files/ceilometer_nova.txt |
|
| 2021-06-01 16:46:09 |
Zakhar Kirpichenko |
description |
Hi!
Ceilometer compute agent fails to get libvirt domain metadata on an Ubuntu 20.04 LTS with the latest updates, kernel 5.4.0-65-generic and Openstack Wallaby Nova compute services installed using the official Wallaby repo for Ubuntu 20.04.
Ceilometer agent is configured with instance_discovery_method = libvirt_metadata. The agent is unable to fetch the domain metadata, and the following error messages appear in /var/log/ceilometer/ceilometer-agent-compute.log on agent start and periodic polling attempts:
2021-06-01 16:01:18.297 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid baf06f57-ac5b-4661-928c-7adaeaea0311 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.298 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 208c0d7a-41a3-4fa6-bf72-2f9594ac6b8d metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.300 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid d979a527-c1ba-4b29-8e30-322d4d2efcf7 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.301 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid a41f21b6-766d-4979-bbe1-84f421b0c3f2 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.302 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid fd5ffe32-c6d6-4898-9ba2-2af1ffebd502 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.302 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid aff042c9-c311-4944-bc42-09ccd5a90eb7 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.303 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 9510bc46-e4e2-490c-9cbe-c9eb5e349b8d metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.304 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 4d2c2c9b-4eff-460a-a00b-19fdbe33f5d4 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster cpu_l3_cache, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.write.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.incoming.packets, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.read.requests, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.outgoing.packets, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.outgoing.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.incoming.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster cpu, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.write.requests, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.307 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.read.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.307 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster memory.usage, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
All domains exist and their metadata is readily available using virsh or a simple Python script. Nova compute service is fully functional, Ceilometer agent is partially functional as it is able to export compute.node.cpu.* metrics but nothing related to libvirt domains.
Installed Ceilometer-related packages:
ceilometer-agent-compute 2:16.0.0-0ubuntu1~cloud0
ceilometer-common 2:16.0.0-0ubuntu1~cloud0
python3-ceilometer 2:16.0.0-0ubuntu1~cloud0
Installed Nova-related packages:
nova-common 3:23.0.0-0ubuntu1~cloud0
nova-compute 3:23.0.0-0ubuntu1~cloud0
nova-compute-kvm 3:23.0.0-0ubuntu1~cloud0
nova-compute-libvirt 3:23.0.0-0ubuntu1~cloud0
python3-nova 3:23.0.0-0ubuntu1~cloud0
python3-novaclient 2:17.4.0-0ubuntu1~cloud0
Installed Libvirt-related packages:
libvirt-clients 6.0.0-0ubuntu8.9
libvirt-daemon 6.0.0-0ubuntu8.9
libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.9
libvirt-daemon-driver-storage-rbd 6.0.0-0ubuntu8.9
libvirt-daemon-system 6.0.0-0ubuntu8.9
libvirt-daemon-system-systemd 6.0.0-0ubuntu8.9
libvirt0:amd64 6.0.0-0ubuntu8.9
python3-libvirt 6.1.0-1
Installed Qemu-related packages:
libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.9
qemu-block-extra:amd64 1:4.2-3ubuntu6.16
qemu-kvm 1:4.2-3ubuntu6.16
qemu-system-common 1:4.2-3ubuntu6.16
qemu-system-data 1:4.2-3ubuntu6.16
qemu-system-gui:amd64 1:4.2-3ubuntu6.16
qemu-system-x86 1:4.2-3ubuntu6.16
qemu-utils 1:4.2-3ubuntu6.16
Apparmor is enabled and running the default configuration, no messages related to apparmor and libvirt, qemu, nova-compute, ceilometer-agent, etc are visible in the logs. I am also attaching the relevant Ceilometer agent and Nova configuration files.
Please let me know if further information is required. |
Hi!
Ceilometer compute agent fails to get libvirt domain metadata on an Ubuntu 20.04 LTS with the latest updates, kernel 5.4.0-65-generic and Openstack Wallaby Nova compute services installed using the official Wallaby repo for Ubuntu 20.04. All components have been deployed manually.
Ceilometer agent is configured with instance_discovery_method = libvirt_metadata. The agent is unable to fetch the domain metadata, and the following error messages appear in /var/log/ceilometer/ceilometer-agent-compute.log on agent start and periodic polling attempts:
2021-06-01 16:01:18.297 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid baf06f57-ac5b-4661-928c-7adaeaea0311 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.298 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 208c0d7a-41a3-4fa6-bf72-2f9594ac6b8d metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.300 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid d979a527-c1ba-4b29-8e30-322d4d2efcf7 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.301 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid a41f21b6-766d-4979-bbe1-84f421b0c3f2 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.302 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid fd5ffe32-c6d6-4898-9ba2-2af1ffebd502 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.302 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid aff042c9-c311-4944-bc42-09ccd5a90eb7 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.303 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 9510bc46-e4e2-490c-9cbe-c9eb5e349b8d metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.304 1835684 ERROR ceilometer.compute.discovery [-] Fail to get domain uuid 4d2c2c9b-4eff-460a-a00b-19fdbe33f5d4 metadata, libvirtError: metadata not found: Requested metadata element is not present: libvirt.libvirtError: metadata not found: Requested metadata element is not present
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster cpu_l3_cache, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.write.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.incoming.packets, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.305 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.read.requests, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.outgoing.packets, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.outgoing.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster network.incoming.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster cpu, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.306 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.write.requests, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.307 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster disk.device.read.bytes, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
2021-06-01 16:01:18.307 1835684 DEBUG ceilometer.polling.manager [-] Skip pollster memory.usage, no resources found this cycle poll_and_notify /usr/lib/python3/dist-packages/ceilometer/polling/manager.py:177
All domains exist and their metadata is readily available using virsh or a simple Python script. Nova compute service is fully functional, Ceilometer agent is partially functional as it is able to export compute.node.cpu.* metrics but nothing related to libvirt domains.
Installed Ceilometer-related packages:
ceilometer-agent-compute 2:16.0.0-0ubuntu1~cloud0
ceilometer-common 2:16.0.0-0ubuntu1~cloud0
python3-ceilometer 2:16.0.0-0ubuntu1~cloud0
Installed Nova-related packages:
nova-common 3:23.0.0-0ubuntu1~cloud0
nova-compute 3:23.0.0-0ubuntu1~cloud0
nova-compute-kvm 3:23.0.0-0ubuntu1~cloud0
nova-compute-libvirt 3:23.0.0-0ubuntu1~cloud0
python3-nova 3:23.0.0-0ubuntu1~cloud0
python3-novaclient 2:17.4.0-0ubuntu1~cloud0
Installed Libvirt-related packages:
libvirt-clients 6.0.0-0ubuntu8.9
libvirt-daemon 6.0.0-0ubuntu8.9
libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.9
libvirt-daemon-driver-storage-rbd 6.0.0-0ubuntu8.9
libvirt-daemon-system 6.0.0-0ubuntu8.9
libvirt-daemon-system-systemd 6.0.0-0ubuntu8.9
libvirt0:amd64 6.0.0-0ubuntu8.9
python3-libvirt 6.1.0-1
Installed Qemu-related packages:
libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.9
qemu-block-extra:amd64 1:4.2-3ubuntu6.16
qemu-kvm 1:4.2-3ubuntu6.16
qemu-system-common 1:4.2-3ubuntu6.16
qemu-system-data 1:4.2-3ubuntu6.16
qemu-system-gui:amd64 1:4.2-3ubuntu6.16
qemu-system-x86 1:4.2-3ubuntu6.16
qemu-utils 1:4.2-3ubuntu6.16
Apparmor is enabled and running the default configuration, no messages related to apparmor and libvirt, qemu, nova-compute, ceilometer-agent, etc are visible in the logs. I am also attaching the relevant Ceilometer agent and Nova configuration files.
Please let me know if further information is required. |
|
| 2021-08-04 12:12:32 |
Aurelien Lourot |
ceilometer: status |
New |
Confirmed |
|
| 2021-10-05 10:00:29 |
Christophe Useinovic |
ceilometer: status |
Confirmed |
In Progress |
|
| 2021-10-05 10:00:29 |
Christophe Useinovic |
ceilometer: assignee |
|
Christophe Useinovic (cuseinovic) |
|
| 2021-10-05 15:20:01 |
Matthias Runge |
ceilometer: importance |
Undecided |
High |
|
| 2021-12-17 13:45:36 |
Corey Bryant |
bug task added |
|
ceilometer (Ubuntu) |
|
| 2021-12-17 13:45:57 |
Corey Bryant |
nominated for series |
|
Ubuntu Hirsute |
|
| 2021-12-17 13:45:57 |
Corey Bryant |
bug task added |
|
ceilometer (Ubuntu Hirsute) |
|
| 2021-12-17 13:45:57 |
Corey Bryant |
nominated for series |
|
Ubuntu Jammy |
|
| 2021-12-17 13:45:57 |
Corey Bryant |
bug task added |
|
ceilometer (Ubuntu Jammy) |
|
| 2021-12-17 13:45:57 |
Corey Bryant |
nominated for series |
|
Ubuntu Impish |
|
| 2021-12-17 13:45:57 |
Corey Bryant |
bug task added |
|
ceilometer (Ubuntu Impish) |
|
| 2021-12-17 13:46:14 |
Corey Bryant |
ceilometer (Ubuntu Hirsute): status |
New |
Confirmed |
|
| 2021-12-17 13:46:18 |
Corey Bryant |
ceilometer (Ubuntu Hirsute): status |
Confirmed |
Triaged |
|
| 2021-12-17 13:46:21 |
Corey Bryant |
ceilometer (Ubuntu Impish): status |
New |
Triaged |
|
| 2021-12-17 13:46:23 |
Corey Bryant |
ceilometer (Ubuntu Jammy): status |
New |
Triaged |
|
| 2021-12-17 13:46:26 |
Corey Bryant |
ceilometer (Ubuntu Impish): importance |
Undecided |
High |
|
| 2021-12-17 13:46:28 |
Corey Bryant |
ceilometer (Ubuntu Hirsute): importance |
Undecided |
High |
|
| 2021-12-17 13:46:32 |
Corey Bryant |
ceilometer (Ubuntu Jammy): importance |
Undecided |
High |
|
| 2021-12-17 16:40:47 |
Gustavo Sanchez |
bug |
|
|
added subscriber Gustavo Sanchez |
| 2021-12-21 17:22:53 |
Thobias Trevisan |
bug |
|
|
added subscriber Thobias Trevisan |
| 2021-12-22 20:10:24 |
Billy Olsen |
bug task added |
|
cloud-archive |
|
| 2021-12-22 20:10:43 |
Billy Olsen |
nominated for series |
|
cloud-archive/wallaby |
|
| 2021-12-22 20:10:43 |
Billy Olsen |
bug task added |
|
cloud-archive/wallaby |
|
| 2021-12-22 20:10:43 |
Billy Olsen |
nominated for series |
|
cloud-archive/xena |
|
| 2021-12-22 20:10:43 |
Billy Olsen |
bug task added |
|
cloud-archive/xena |
|
| 2021-12-22 20:10:52 |
Billy Olsen |
cloud-archive/wallaby: status |
New |
Triaged |
|
| 2021-12-22 20:10:54 |
Billy Olsen |
cloud-archive/xena: status |
New |
Triaged |
|
| 2021-12-22 20:11:00 |
Billy Olsen |
cloud-archive: status |
New |
Triaged |
|
| 2021-12-22 20:32:37 |
Billy Olsen |
bug |
|
|
added subscriber Canonical Field High |
| 2022-01-26 21:55:32 |
Brian Murray |
ceilometer (Ubuntu Hirsute): status |
Triaged |
Won't Fix |
|
| 2022-02-10 10:48:40 |
OpenStack Infra |
cloud-archive/xena: status |
Triaged |
Fix Released |
|
| 2022-02-15 09:31:03 |
James Page |
nominated for series |
|
cloud-archive/yoga |
|
| 2022-02-15 09:31:03 |
James Page |
bug task added |
|
cloud-archive/yoga |
|
| 2022-02-15 10:22:05 |
James Page |
cloud-archive/wallaby: importance |
Undecided |
High |
|
| 2022-02-15 10:22:10 |
James Page |
cloud-archive/xena: importance |
Undecided |
High |
|
| 2022-02-15 10:22:13 |
James Page |
cloud-archive/xena: status |
Fix Released |
Triaged |
|
| 2022-02-15 10:22:15 |
James Page |
cloud-archive/yoga: importance |
Undecided |
High |
|
| 2022-03-08 12:43:31 |
OpenStack Infra |
cloud-archive/yoga: status |
Triaged |
Fix Released |
|
| 2022-03-21 10:57:11 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2022-03-27 10:13:25 |
XiaoRuiguo |
bug |
|
|
added subscriber XiaoRuiguo |
| 2022-05-23 18:41:18 |
Corey Bryant |
ceilometer (Ubuntu Jammy): status |
Triaged |
Fix Released |
|
| 2022-05-23 18:47:24 |
Corey Bryant |
ceilometer (Ubuntu Impish): status |
Triaged |
Fix Released |
|
| 2022-05-23 18:47:29 |
Corey Bryant |
cloud-archive/xena: status |
Triaged |
Fix Released |
|
| 2022-05-23 18:47:32 |
Corey Bryant |
cloud-archive/wallaby: status |
Triaged |
Fix Released |
|
| 2022-05-23 18:47:37 |
Corey Bryant |
ceilometer (Ubuntu): status |
Triaged |
Fix Released |
|
| 2022-07-04 11:30:39 |
James Page |
removed subscriber Canonical Field High |
|
|
|
