kolla-ansible

nova-compute ComputeHostNotFound_Remote on a new kolla-ansible deployment

Bug #1718836 reported by Imran Niazi on 2017-09-22

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	kolla-ansible	Invalid	Undecided	Unassigned

Bug Description

Deployed Openstack multinode 5.0.0 on centos 7.4. external ceph. 1 network, 2 controllers, and 3 computes nodes (with nova-compute deployed on controllers as well).

After deployment, "openstack hypervisor list" is empty.

openstack compute service list:
[root@metroplex kolla]# openstack compute service list
+----+------------------+---------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+---------------+----------+---------+-------+----------------------------+
| 4 | nova-scheduler | os-control-01 | internal | enabled | up | 2017-09-22T04:04:14.000000 |
| 8 | nova-scheduler | os-control-02 | internal | enabled | up | 2017-09-22T04:04:15.000000 |
| 10 | nova-conductor | os-control-01 | internal | enabled | up | 2017-09-22T04:04:17.000000 |
| 16 | nova-conductor | os-control-02 | internal | enabled | up | 2017-09-22T04:04:09.000000 |
| 20 | nova-consoleauth | os-control-01 | internal | enabled | up | 2017-09-22T04:04:13.000000 |
| 22 | nova-consoleauth | os-control-02 | internal | enabled | up | 2017-09-22T04:04:15.000000 |
| 30 | nova-compute | os-compute-03 | nova | enabled | up | 2017-09-22T04:04:13.000000 |
| 42 | nova-compute | os-compute-01 | nova | enabled | up | 2017-09-22T04:04:11.000000 |
| 44 | nova-compute | os-compute-02 | nova | enabled | up | 2017-09-22T04:04:09.000000 |
| 46 | nova-compute | os-control-02 | nova | enabled | up | 2017-09-22T04:04:10.000000 |
| 48 | nova-compute | os-control-01 | nova | enabled | up | 2017-09-22T04:04:10.000000 |
+----+------------------+---------------+----------+---------+-------+----------------------------+

See this in /var/lib/docker/volumes/kolla_logs/_data/nova/nova-compute.log on all the hosts running nova-compute

2017-09-21 21:05:51.370 7 ERROR nova.compute.manager [req-8f83bf82-43f9-4d42-8e72-19238fb99768 - - - - -] No compute node record for host os-compute-01: ComputeHostNotFound_Remote: Compute host os-compute-01 could not be found.
Traceback (most recent call last):

File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/conductor/manager.py", line 123, in _object_dispatch
return getattr(target, method)(*args, **kwargs)

File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
result = fn(cls, context, *args, **kwargs)

File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/compute_node.py", line 437, in get_all_by_host
use_slave=use_slave)

File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 235, in wrapper
return f(*args, **kwargs)

File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/objects/compute_node.py", line 432, in _db_compute_node_get_all_by_host
return db.compute_node_get_all_by_host(context, host)

File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/api.py", line 297, in compute_node_get_all_by_host
return IMPL.compute_node_get_all_by_host(context, host)

File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 280, in wrapped
return f(context, *args, **kwargs)

File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 713, in compute_node_get_all_by_host
raise exception.ComputeHostNotFound(host=host)

ComputeHostNotFound: Compute host os-compute-01 could not be found.
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager [req-8f83bf82-43f9-4d42-8e72-19238fb99768 - - - - -] Error updating resources for node os-compute-01.: Error: error calling conf_read_file: error code 22
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager Traceback (most recent call last):
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 6629, in update_available_resource_for_node
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager rt.update_available_resource(context, nodename)
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 641, in update_available_resource
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager resources = self.driver.get_available_resource(nodename)
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5769, in get_available_resource
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager disk_info_dict = self._get_local_gb_info()
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5336, in _get_local_gb_info
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager info = LibvirtDriver._get_rbd_driver().get_pool_info()
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 369, in get_pool_info
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager with RADOSClient(self) as client:
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 103, in __init__
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager self.cluster, self.ioctx = driver._connect_to_rados(pool)
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 132, in _connect_to_rados
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager conffile=self.ceph_conf)
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "rados.pyx", line 525, in rados.Rados.__init__ (rados.c:5719)
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "rados.pyx", line 568, in rados.Rados.__setup (rados.c:6448)
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager File "rados.pyx", line 631, in rados.Rados.conf_read_file (rados.c:7384)
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager Error: error calling conf_read_file: error code 22
2017-09-21 21:05:51.374 7 ERROR nova.compute.manager

[root@os-compute-01 kolla]# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 26
Model name: Intel(R) Xeon(R) CPU L5520 @ 2.27GHz
Stepping: 5
CPU MHz: 1600.000
CPU max MHz: 2267.0000
CPU min MHz: 1600.0000
BogoMIPS: 4533.39
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-3,8-11
NUMA node1 CPU(s): 4-7,12-15
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm tpr_shadow vnmi flexpriority ept vpid dtherm ida

Imran Niazi (iniazi) on 2017-09-22

affects:

kolla → kolla-ansible

Revision history for this message

Imran Niazi (iniazi) wrote on 2017-09-22:

Also, built the docker images via source and my globals.yml:

[root@metroplex kolla]# grep -v "^\s*#" /etc/kolla/globals.yml | uniq

kolla_base_distro: "centos"

kolla_install_type: "source"

kolla_internal_vip_address: "x.x.x.x"

kolla_internal_fqdn: "control.DOMAIN"

kolla_external_fqdn: "control.DOMAIN"

docker_registry: "10.0.0.32:5000"

network_interface: "team0"

tunnel_interface: "team1"

neutron_external_interface: "team2"

keepalived_virtual_router_id: "97"

enable_aodh: "yes"
enable_barbican: "yes"
enable_ceilometer: "yes"
enable_cinder: "yes"
enable_collectd: "yes"
enable_congress: "yes"
enable_designate: "yes"
enable_etcd: "yes"
enable_freezer: "yes"
enable_gnocchi: "yes"
enable_grafana: "no"
enable_karbor: "yes"
enable_kuryr: "yes"
enable_magnum: "yes"
enable_manila: "yes"
enable_manila_backend_generic: "yes"
enable_mistral: "yes"
enable_murayes: "no"
enable_neutron_lbaas: "yes"
enable_neutron_fwaas: "yes"
enable_neutron_vpnaas: "yes"
enable_octavia: "yes"
enable_ovs_dpdk: "no"
enable_panko: "yes"
enable_redis: "yes"
enable_sahara: "yes"
enable_senlin: "yes"
enable_telegraf: "yes"
enable_trove: "yes"
enable_watcher: "yes"
enable_zun: "yes"

glance_backend_ceph: "yes"

gnocchi_backend_storage: "yes"

cinder_backend_ceph: "yes"

cinder_backup_driver: "ceph"

designate_ns_record: "node.DOMAIN"

nova_backend_ceph: "ceph"

tempest_image_id:
tempest_flavor_ref_id:
tempest_public_network_id:
tempest_floating_network_name:

Revision history for this message

Imran Niazi (iniazi) wrote on 2017-09-30:

This was a non-issue. typo in globals.yml for enabling ceph.

Changed in kolla-ansible:
status:	New → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.