Detailed bug description: I found a lot of ERROR logs for Ceilometer that it cannot obtain IP address of instance:
2016-04-28 09:03:06.254 14501 ERROR ceilometer.hardware.discovery [req-fc590e4d-3e11-48d3-98f7-ee94af3c43ec admin - - - -] Couldn't obtain IP address of instance 5a4c7651-7d3e-4bab-b8ee-fdda9be15703 ... 2016-04-28 09:42:55.889 14501 ERROR ceilometer.hardware.discovery [req-fc590e4d-3e11-48d3-98f7-ee94af3c43ec admin - - - -] Couldn't obtain IP address of instance bfd24135-e279-4b7f-be7a-1aadf593db04
Meanwhile I see no problem with mentioned instances, they are up and running.
(.venv) [root@fuel work]# nova show 5a4c7651-7d3e-4bab-b8ee-fdda9be15703 +--------------------------------------+----------------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------------+ | OS-DCF:diskConfig | AUTO | | OS-EXT-AZ:availability_zone | nova | | OS-EXT-SRV-ATTR:host | node-54.domain.tld | | OS-EXT-SRV-ATTR:hypervisor_hostname | node-54.domain.tld | | OS-EXT-SRV-ATTR:instance_name | instance-000010bc | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2016-04-27T17:03:41.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | admin_internal_net network | 192.168.108.74 | | config_drive | | | created | 2016-04-27T16:58:28Z | | flavor | gig (e8c39500-0a22-4c25-afe2-f8ed15f61e3d) | | hostId | 0731685cfe27ffd7fbfb59b032d8496150e63e660e8c2bfc11496a48 | | id | 5a4c7651-7d3e-4bab-b8ee-fdda9be15703 | | image | Xenial (9355f643-f72b-4e6f-83cb-124b706ec87e) | | key_name | - | | metadata | {} | | name | StressCPU-7 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | ACTIVE | | tenant_id | d41dac66d080416ebbd597e0793e5aca | | updated | 2016-04-27T17:03:41Z | | user_id | 2a9a8a83ab63483299b69a8f81dbdf35 | +--------------------------------------+----------------------------------------------------------+
Steps to reproduce: 1. Boot ~20-30 VMs from image (ephemeral volume is used) 2. Keep them up and running for several hours 3. Check logs
Expected results: 1. No ERRORs from ceilometer-polling
Actual result: 1. I see mentioned log entries
Reproducibility: 100%
Workaround: N/A
Impact: Unknown. I do not know what it leads to
Description of the environment: * 10 baremetal nodes: - CPU: 12 x 2.10 GHz - Disks: 2 drives (SSD - 80 GB, HDD - 931.5 GB), 1006.0 GB total - Memory: 2 x 16.0 GB, 32.0 GB total - NUMA topology: 1 NUMA node * Node roles: - 1 ElasticSearch / Kibana node - 1 InfluxDB / Grafana node - 3 controllers (1 was is offline because of disk problems) - 5 computes * Details: - OS: Mitaka on Ubuntu 14.04 - Compute: KVM - Neutron with VLAN segmentation - Ceph RBD for volumes (Cinder) - Ceph RadosGW for objects (Swift API) - Ceph RBD for ephemeral volumes (Nova) - Ceph RBD for images (Glance)
Additional information: Diagnostic snapshot: http://mos-scale-share.mirantis.com/env14/fuel-snapshot-2016-04-28_07-46-04.tar.xz
Detailed bug description:
I found a lot of ERROR logs for Ceilometer that it cannot obtain IP address of instance:
2016-04-28 09:03:06.254 14501 ERROR ceilometer. hardware. discovery [req-fc590e4d- 3e11-48d3- 98f7-ee94af3c43 ec admin - - - -] Couldn't obtain IP address of instance 5a4c7651- 7d3e-4bab- b8ee-fdda9be157 03 hardware. discovery [req-fc590e4d- 3e11-48d3- 98f7-ee94af3c43 ec admin - - - -] Couldn't obtain IP address of instance bfd24135- e279-4b7f- be7a-1aadf593db 04
...
2016-04-28 09:42:55.889 14501 ERROR ceilometer.
Meanwhile I see no problem with mentioned instances, they are up and running.
(.venv) [root@fuel work]# nova show 5a4c7651- 7d3e-4bab- b8ee-fdda9be157 03 ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- ------- ------- + ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- ------- ------- + AZ:availability _zone | nova | SRV-ATTR: host | node-54.domain.tld | SRV-ATTR: hypervisor_ hostname | node-54.domain.tld | SRV-ATTR: instance_ name | instance-000010bc | STS:power_ state | 1 | STS:task_ state | - | USG:launched_ at | 2016-04- 27T17:03: 41.000000 | USG:terminated_ at | - | 27T16:58: 28Z | 0a22-4c25- afe2-f8ed15f61e 3d) | 7fbfb59b032d849 6150e63e660e8c2 bfc11496a48 | 7d3e-4bab- b8ee-fdda9be157 03 | f72b-4e6f- 83cb-124b706ec8 7e) | volumes: volumes_ attached | [] | ebbd597e0793e5a ca | 27T17:03: 41Z | 299b69a8f81dbdf 35 | ------- ------- ------- ------- ----+-- ------- ------- ------- ------- ------- ------- ------- ------- +
+------
| Property | Value |
+------
| OS-DCF:diskConfig | AUTO |
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-
| OS-EXT-STS:vm_state | active |
| OS-SRV-
| OS-SRV-
| accessIPv4 | |
| accessIPv6 | |
| admin_internal_net network | 192.168.108.74 |
| config_drive | |
| created | 2016-04-
| flavor | gig (e8c39500-
| hostId | 0731685cfe27ffd
| id | 5a4c7651-
| image | Xenial (9355f643-
| key_name | - |
| metadata | {} |
| name | StressCPU-7 |
| os-extended-
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | d41dac66d080416
| updated | 2016-04-
| user_id | 2a9a8a83ab63483
+------
Steps to reproduce:
1. Boot ~20-30 VMs from image (ephemeral volume is used)
2. Keep them up and running for several hours
3. Check logs
Expected results:
1. No ERRORs from ceilometer-polling
Actual result:
1. I see mentioned log entries
Reproducibility:
100%
Workaround:
N/A
Impact:
Unknown. I do not know what it leads to
Description of the environment:
* 10 baremetal nodes:
- CPU: 12 x 2.10 GHz
- Disks: 2 drives (SSD - 80 GB, HDD - 931.5 GB), 1006.0 GB total
- Memory: 2 x 16.0 GB, 32.0 GB total
- NUMA topology: 1 NUMA node
* Node roles:
- 1 ElasticSearch / Kibana node
- 1 InfluxDB / Grafana node
- 3 controllers (1 was is offline because of disk problems)
- 5 computes
* Details:
- OS: Mitaka on Ubuntu 14.04
- Compute: KVM
- Neutron with VLAN segmentation
- Ceph RBD for volumes (Cinder)
- Ceph RadosGW for objects (Swift API)
- Ceph RBD for ephemeral volumes (Nova)
- Ceph RBD for images (Glance)
Additional information: mos-scale- share.mirantis. com/env14/ fuel-snapshot- 2016-04- 28_07-46- 04.tar. xz
Diagnostic snapshot: http://