| 2017-10-26 12:13:06 |
Khawar Munir Abbasi |
bug |
|
|
added bug |
| 2017-10-26 12:14:14 |
Khawar Munir Abbasi |
description |
Hi,
I upgraded OpenStack Kolla Newton to Ocata recently. Everything works fine except creating new VMs with small.m1 flavor size. I can one more cirros VM with tiny.m1 flavor. This means its most likely the problem of space. I checked the logs of neutron,nova and glance. Only nova-conductor logs show error. But that is too generic "No valid host was found". I've investigated that if i somehow increase value of "disk_available_least" in hypervisor then i will create new VMs. Here you can see the relevant details:
nova conductor logs
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to schedule instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 868, in schedule_and_build_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 599, in _schedule_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=self.retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise result
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise exception.NoValidHost(reason=reason)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.486 19 WARNING nova.scheduler.utils [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to compute_task_build_instances: No valid host was found. $
Traceback (most recent call last):
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
return func(*args, **kwargs)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0b6d0bae68d kolla/ubuntu-source-horizon:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks horizon
073f62b32fd1 kolla/ubuntu-source-tacker:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks tacker
af07c296fba1 kolla/ubuntu-source-cron:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks cron
e81b3b76283a kolla/ubuntu-source-mongodb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mongodb
bb0217551442 kolla/ubuntu-source-heat-engine:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_engine
bedfaa812c7a kolla/ubuntu-source-heat-api-cfn:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks heat_api_cfn
669316195ccb kolla/ubuntu-source-heat-api:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_api
b4071a7ed7a7 kolla/ubuntu-source-neutron-metadata-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_metadata_agent
d1cc9cce028e kolla/ubuntu-source-neutron-l3-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_l3_agent
f0aa10f68d96 kolla/ubuntu-source-neutron-dhcp-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_dhcp_agent
6e0a605e7326 kolla/ubuntu-source-neutron-openvswitch-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_openvswitch_agent
f0d4060b71c3 kolla/ubuntu-source-neutron-server:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_server
54a85670ee1b kolla/ubuntu-source-openvswitch-vswitchd:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks openvswitch_vswitchd
13e0f8820afe kolla/ubuntu-source-openvswitch-db-server:4.0.3 "kolla_start" 2 weeks ago Up 17 hours openvswitch_db
fb59adc87585 kolla/ubuntu-source-nova-compute:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_compute
dd76f09e8b41 kolla/ubuntu-source-nova-novncproxy:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_novncproxy
dbb563f6539b kolla/ubuntu-source-nova-consoleauth:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_consoleauth
a2b05c11fe53 kolla/ubuntu-source-nova-conductor:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_conductor
d4783aaff3ee kolla/ubuntu-source-nova-scheduler:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_scheduler
9e1b532d2fcf kolla/ubuntu-source-nova-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_api
08056288cc13 kolla/ubuntu-source-nova-placement-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours placement_api
71a75282a7e6 kolla/ubuntu-source-nova-libvirt:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_libvirt
91788c4b6e18 kolla/ubuntu-source-nova-ssh:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_ssh
4b3e75902d51 kolla/ubuntu-source-glance-registry:4.0.3 "kolla_start" 2 weeks ago Up 17 hours glance_registry
c88023bebdb4 kolla/ubuntu-source-glance-api:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks glance_api
ea150c5b1353 kolla/ubuntu-source-keystone:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks keystone
ad1633c5c3ac kolla/ubuntu-source-rabbitmq:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks rabbitmq
58afd39b647a kolla/ubuntu-source-mariadb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mariadb
9b084133b77a kolla/ubuntu-source-kolla-toolbox:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks kolla_toolbox
2fb94fc409a8 kolla/ubuntu-source-fluentd:4.0.3 "kolla_start" 2 weeks ago Up 17 hours fluentd
8504e5e0879f kolla/ubuntu-source-memcached:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks memcached
nova hypervisor-show xxxxx
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
| cpu_info_arch | x86_64 |
| cpu_info_features | ["pge", "avx", "xsaveopt", "clflush", |
| | "sep", "rtm", "tsc_adjust", "vme", |
| | "dtes64", "invpcid", "msr", "sse", |
| | "xsave", "smap", "vmx", "erms", "xtpr", |
| | "cmov", "hle", "smep", "nx", "est", |
| | "pat", "monitor", "smx", "pbe", "lm", |
| | "tsc", "adx", "3dnowprefetch", "fpu", |
| | "syscall", "tm", "sse4.1", "fxsr", |
| | "sse4.2", "pclmuldq", "pcid", "pae", |
| | "pni", "tsc-deadline", "mmx", "osxsave", |
| | "cx8", "mce", "de", "tm2", "ht", "dca", |
| | "lahf_lm", "abm", "rdseed", "popcnt", |
| | "mca", "pdpe1gb", "apic", "fsgsbase", |
| | "f16c", "pse", "ds", "invtsc", |
| | "mbm_total", "aes", "avx2", "sse2", |
| | "ss", "ds_cpl", "arat", "bmi1", "bmi2", |
| | "acpi", "ssse3", "rdtscp", "mbm_local", |
| | "cmt", "cx16", "pse36", "mtrr", "movbe", |
| | "pdcm", "fma", "rdrand", "x2apic"] |
| cpu_info_model | Broadwell |
| cpu_info_topology_cells | 2 |
| cpu_info_topology_cores | 22 |
| cpu_info_topology_sockets | 1 |
| cpu_info_topology_threads | 2 |
| cpu_info_vendor | Intel |
| current_workload | 0 |
| disk_available_least | 2 |
| free_disk_gb | 271 |
| free_ram_mb | 49461 |
| host_ip | xxxxxx |
| hypervisor_hostname | xxxxxxx |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 1 |
| local_gb | 394 |
| local_gb_used | 184 |
| memory_mb | 64309 |
| memory_mb_used | 27552 |
| running_vms | 4 |
| service_disabled_reason | None |
| service_host | xxxxxx |
| service_id | 11 |
| state | up |
| status | enabled |
| vcpus | 88 |
| vcpus_used | 8 |
+---------------------------+------------------------------------------+ |
Hi,
I upgraded OpenStack Kolla Newton to Ocata recently. Everything works fine except creating now new VMs with small.m1 flavors or larger than this. I can one more cirros VM with tiny.m1 flavor. This means its most likely the problem of space. I checked the logs of neutron,nova and glance. Only nova-conductor logs show error. But that is too generic "No valid host was found". I've investigated that if i somehow increase value of "disk_available_least" in hypervisor then i will create new VMs. Here you can see the relevant details:
nova conductor logs
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to schedule instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 868, in schedule_and_build_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 599, in _schedule_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=self.retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise result
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise exception.NoValidHost(reason=reason)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.486 19 WARNING nova.scheduler.utils [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to compute_task_build_instances: No valid host was found. $
Traceback (most recent call last):
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
return func(*args, **kwargs)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0b6d0bae68d kolla/ubuntu-source-horizon:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks horizon
073f62b32fd1 kolla/ubuntu-source-tacker:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks tacker
af07c296fba1 kolla/ubuntu-source-cron:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks cron
e81b3b76283a kolla/ubuntu-source-mongodb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mongodb
bb0217551442 kolla/ubuntu-source-heat-engine:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_engine
bedfaa812c7a kolla/ubuntu-source-heat-api-cfn:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks heat_api_cfn
669316195ccb kolla/ubuntu-source-heat-api:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_api
b4071a7ed7a7 kolla/ubuntu-source-neutron-metadata-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_metadata_agent
d1cc9cce028e kolla/ubuntu-source-neutron-l3-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_l3_agent
f0aa10f68d96 kolla/ubuntu-source-neutron-dhcp-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_dhcp_agent
6e0a605e7326 kolla/ubuntu-source-neutron-openvswitch-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_openvswitch_agent
f0d4060b71c3 kolla/ubuntu-source-neutron-server:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_server
54a85670ee1b kolla/ubuntu-source-openvswitch-vswitchd:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks openvswitch_vswitchd
13e0f8820afe kolla/ubuntu-source-openvswitch-db-server:4.0.3 "kolla_start" 2 weeks ago Up 17 hours openvswitch_db
fb59adc87585 kolla/ubuntu-source-nova-compute:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_compute
dd76f09e8b41 kolla/ubuntu-source-nova-novncproxy:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_novncproxy
dbb563f6539b kolla/ubuntu-source-nova-consoleauth:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_consoleauth
a2b05c11fe53 kolla/ubuntu-source-nova-conductor:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_conductor
d4783aaff3ee kolla/ubuntu-source-nova-scheduler:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_scheduler
9e1b532d2fcf kolla/ubuntu-source-nova-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_api
08056288cc13 kolla/ubuntu-source-nova-placement-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours placement_api
71a75282a7e6 kolla/ubuntu-source-nova-libvirt:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_libvirt
91788c4b6e18 kolla/ubuntu-source-nova-ssh:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_ssh
4b3e75902d51 kolla/ubuntu-source-glance-registry:4.0.3 "kolla_start" 2 weeks ago Up 17 hours glance_registry
c88023bebdb4 kolla/ubuntu-source-glance-api:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks glance_api
ea150c5b1353 kolla/ubuntu-source-keystone:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks keystone
ad1633c5c3ac kolla/ubuntu-source-rabbitmq:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks rabbitmq
58afd39b647a kolla/ubuntu-source-mariadb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mariadb
9b084133b77a kolla/ubuntu-source-kolla-toolbox:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks kolla_toolbox
2fb94fc409a8 kolla/ubuntu-source-fluentd:4.0.3 "kolla_start" 2 weeks ago Up 17 hours fluentd
8504e5e0879f kolla/ubuntu-source-memcached:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks memcached
nova hypervisor-show xxxxx
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
| cpu_info_arch | x86_64 |
| cpu_info_features | ["pge", "avx", "xsaveopt", "clflush", |
| | "sep", "rtm", "tsc_adjust", "vme", |
| | "dtes64", "invpcid", "msr", "sse", |
| | "xsave", "smap", "vmx", "erms", "xtpr", |
| | "cmov", "hle", "smep", "nx", "est", |
| | "pat", "monitor", "smx", "pbe", "lm", |
| | "tsc", "adx", "3dnowprefetch", "fpu", |
| | "syscall", "tm", "sse4.1", "fxsr", |
| | "sse4.2", "pclmuldq", "pcid", "pae", |
| | "pni", "tsc-deadline", "mmx", "osxsave", |
| | "cx8", "mce", "de", "tm2", "ht", "dca", |
| | "lahf_lm", "abm", "rdseed", "popcnt", |
| | "mca", "pdpe1gb", "apic", "fsgsbase", |
| | "f16c", "pse", "ds", "invtsc", |
| | "mbm_total", "aes", "avx2", "sse2", |
| | "ss", "ds_cpl", "arat", "bmi1", "bmi2", |
| | "acpi", "ssse3", "rdtscp", "mbm_local", |
| | "cmt", "cx16", "pse36", "mtrr", "movbe", |
| | "pdcm", "fma", "rdrand", "x2apic"] |
| cpu_info_model | Broadwell |
| cpu_info_topology_cells | 2 |
| cpu_info_topology_cores | 22 |
| cpu_info_topology_sockets | 1 |
| cpu_info_topology_threads | 2 |
| cpu_info_vendor | Intel |
| current_workload | 0 |
| disk_available_least | 2 |
| free_disk_gb | 271 |
| free_ram_mb | 49461 |
| host_ip | xxxxxx |
| hypervisor_hostname | xxxxxxx |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 1 |
| local_gb | 394 |
| local_gb_used | 184 |
| memory_mb | 64309 |
| memory_mb_used | 27552 |
| running_vms | 4 |
| service_disabled_reason | None |
| service_host | xxxxxx |
| service_id | 11 |
| state | up |
| status | enabled |
| vcpus | 88 |
| vcpus_used | 8 |
+---------------------------+------------------------------------------+ |
|
| 2017-10-26 12:14:31 |
Khawar Munir Abbasi |
description |
Hi,
I upgraded OpenStack Kolla Newton to Ocata recently. Everything works fine except creating now new VMs with small.m1 flavors or larger than this. I can one more cirros VM with tiny.m1 flavor. This means its most likely the problem of space. I checked the logs of neutron,nova and glance. Only nova-conductor logs show error. But that is too generic "No valid host was found". I've investigated that if i somehow increase value of "disk_available_least" in hypervisor then i will create new VMs. Here you can see the relevant details:
nova conductor logs
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to schedule instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 868, in schedule_and_build_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 599, in _schedule_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=self.retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise result
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise exception.NoValidHost(reason=reason)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.486 19 WARNING nova.scheduler.utils [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to compute_task_build_instances: No valid host was found. $
Traceback (most recent call last):
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
return func(*args, **kwargs)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0b6d0bae68d kolla/ubuntu-source-horizon:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks horizon
073f62b32fd1 kolla/ubuntu-source-tacker:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks tacker
af07c296fba1 kolla/ubuntu-source-cron:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks cron
e81b3b76283a kolla/ubuntu-source-mongodb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mongodb
bb0217551442 kolla/ubuntu-source-heat-engine:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_engine
bedfaa812c7a kolla/ubuntu-source-heat-api-cfn:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks heat_api_cfn
669316195ccb kolla/ubuntu-source-heat-api:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_api
b4071a7ed7a7 kolla/ubuntu-source-neutron-metadata-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_metadata_agent
d1cc9cce028e kolla/ubuntu-source-neutron-l3-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_l3_agent
f0aa10f68d96 kolla/ubuntu-source-neutron-dhcp-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_dhcp_agent
6e0a605e7326 kolla/ubuntu-source-neutron-openvswitch-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_openvswitch_agent
f0d4060b71c3 kolla/ubuntu-source-neutron-server:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_server
54a85670ee1b kolla/ubuntu-source-openvswitch-vswitchd:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks openvswitch_vswitchd
13e0f8820afe kolla/ubuntu-source-openvswitch-db-server:4.0.3 "kolla_start" 2 weeks ago Up 17 hours openvswitch_db
fb59adc87585 kolla/ubuntu-source-nova-compute:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_compute
dd76f09e8b41 kolla/ubuntu-source-nova-novncproxy:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_novncproxy
dbb563f6539b kolla/ubuntu-source-nova-consoleauth:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_consoleauth
a2b05c11fe53 kolla/ubuntu-source-nova-conductor:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_conductor
d4783aaff3ee kolla/ubuntu-source-nova-scheduler:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_scheduler
9e1b532d2fcf kolla/ubuntu-source-nova-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_api
08056288cc13 kolla/ubuntu-source-nova-placement-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours placement_api
71a75282a7e6 kolla/ubuntu-source-nova-libvirt:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_libvirt
91788c4b6e18 kolla/ubuntu-source-nova-ssh:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_ssh
4b3e75902d51 kolla/ubuntu-source-glance-registry:4.0.3 "kolla_start" 2 weeks ago Up 17 hours glance_registry
c88023bebdb4 kolla/ubuntu-source-glance-api:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks glance_api
ea150c5b1353 kolla/ubuntu-source-keystone:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks keystone
ad1633c5c3ac kolla/ubuntu-source-rabbitmq:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks rabbitmq
58afd39b647a kolla/ubuntu-source-mariadb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mariadb
9b084133b77a kolla/ubuntu-source-kolla-toolbox:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks kolla_toolbox
2fb94fc409a8 kolla/ubuntu-source-fluentd:4.0.3 "kolla_start" 2 weeks ago Up 17 hours fluentd
8504e5e0879f kolla/ubuntu-source-memcached:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks memcached
nova hypervisor-show xxxxx
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
| cpu_info_arch | x86_64 |
| cpu_info_features | ["pge", "avx", "xsaveopt", "clflush", |
| | "sep", "rtm", "tsc_adjust", "vme", |
| | "dtes64", "invpcid", "msr", "sse", |
| | "xsave", "smap", "vmx", "erms", "xtpr", |
| | "cmov", "hle", "smep", "nx", "est", |
| | "pat", "monitor", "smx", "pbe", "lm", |
| | "tsc", "adx", "3dnowprefetch", "fpu", |
| | "syscall", "tm", "sse4.1", "fxsr", |
| | "sse4.2", "pclmuldq", "pcid", "pae", |
| | "pni", "tsc-deadline", "mmx", "osxsave", |
| | "cx8", "mce", "de", "tm2", "ht", "dca", |
| | "lahf_lm", "abm", "rdseed", "popcnt", |
| | "mca", "pdpe1gb", "apic", "fsgsbase", |
| | "f16c", "pse", "ds", "invtsc", |
| | "mbm_total", "aes", "avx2", "sse2", |
| | "ss", "ds_cpl", "arat", "bmi1", "bmi2", |
| | "acpi", "ssse3", "rdtscp", "mbm_local", |
| | "cmt", "cx16", "pse36", "mtrr", "movbe", |
| | "pdcm", "fma", "rdrand", "x2apic"] |
| cpu_info_model | Broadwell |
| cpu_info_topology_cells | 2 |
| cpu_info_topology_cores | 22 |
| cpu_info_topology_sockets | 1 |
| cpu_info_topology_threads | 2 |
| cpu_info_vendor | Intel |
| current_workload | 0 |
| disk_available_least | 2 |
| free_disk_gb | 271 |
| free_ram_mb | 49461 |
| host_ip | xxxxxx |
| hypervisor_hostname | xxxxxxx |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 1 |
| local_gb | 394 |
| local_gb_used | 184 |
| memory_mb | 64309 |
| memory_mb_used | 27552 |
| running_vms | 4 |
| service_disabled_reason | None |
| service_host | xxxxxx |
| service_id | 11 |
| state | up |
| status | enabled |
| vcpus | 88 |
| vcpus_used | 8 |
+---------------------------+------------------------------------------+ |
Hi,
I upgraded OpenStack Kolla Newton to Ocata recently. Everything works fine except creating new VMs with small.m1 flavors or larger than this. I can one more cirros VM with tiny.m1 flavor. This means its most likely the problem of space. I checked the logs of neutron,nova and glance. Only nova-conductor logs show error. But that is too generic "No valid host was found". I've investigated that if i somehow increase value of "disk_available_least" in hypervisor then i will create new VMs. Here you can see the relevant details:
nova conductor logs
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to schedule instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 868, in schedule_and_build_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 599, in _schedule_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=self.retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise result
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise exception.NoValidHost(reason=reason)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.486 19 WARNING nova.scheduler.utils [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to compute_task_build_instances: No valid host was found. $
Traceback (most recent call last):
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
return func(*args, **kwargs)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0b6d0bae68d kolla/ubuntu-source-horizon:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks horizon
073f62b32fd1 kolla/ubuntu-source-tacker:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks tacker
af07c296fba1 kolla/ubuntu-source-cron:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks cron
e81b3b76283a kolla/ubuntu-source-mongodb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mongodb
bb0217551442 kolla/ubuntu-source-heat-engine:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_engine
bedfaa812c7a kolla/ubuntu-source-heat-api-cfn:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks heat_api_cfn
669316195ccb kolla/ubuntu-source-heat-api:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_api
b4071a7ed7a7 kolla/ubuntu-source-neutron-metadata-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_metadata_agent
d1cc9cce028e kolla/ubuntu-source-neutron-l3-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_l3_agent
f0aa10f68d96 kolla/ubuntu-source-neutron-dhcp-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_dhcp_agent
6e0a605e7326 kolla/ubuntu-source-neutron-openvswitch-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_openvswitch_agent
f0d4060b71c3 kolla/ubuntu-source-neutron-server:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_server
54a85670ee1b kolla/ubuntu-source-openvswitch-vswitchd:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks openvswitch_vswitchd
13e0f8820afe kolla/ubuntu-source-openvswitch-db-server:4.0.3 "kolla_start" 2 weeks ago Up 17 hours openvswitch_db
fb59adc87585 kolla/ubuntu-source-nova-compute:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_compute
dd76f09e8b41 kolla/ubuntu-source-nova-novncproxy:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_novncproxy
dbb563f6539b kolla/ubuntu-source-nova-consoleauth:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_consoleauth
a2b05c11fe53 kolla/ubuntu-source-nova-conductor:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_conductor
d4783aaff3ee kolla/ubuntu-source-nova-scheduler:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_scheduler
9e1b532d2fcf kolla/ubuntu-source-nova-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_api
08056288cc13 kolla/ubuntu-source-nova-placement-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours placement_api
71a75282a7e6 kolla/ubuntu-source-nova-libvirt:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_libvirt
91788c4b6e18 kolla/ubuntu-source-nova-ssh:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_ssh
4b3e75902d51 kolla/ubuntu-source-glance-registry:4.0.3 "kolla_start" 2 weeks ago Up 17 hours glance_registry
c88023bebdb4 kolla/ubuntu-source-glance-api:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks glance_api
ea150c5b1353 kolla/ubuntu-source-keystone:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks keystone
ad1633c5c3ac kolla/ubuntu-source-rabbitmq:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks rabbitmq
58afd39b647a kolla/ubuntu-source-mariadb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mariadb
9b084133b77a kolla/ubuntu-source-kolla-toolbox:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks kolla_toolbox
2fb94fc409a8 kolla/ubuntu-source-fluentd:4.0.3 "kolla_start" 2 weeks ago Up 17 hours fluentd
8504e5e0879f kolla/ubuntu-source-memcached:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks memcached
nova hypervisor-show xxxxx
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
| cpu_info_arch | x86_64 |
| cpu_info_features | ["pge", "avx", "xsaveopt", "clflush", |
| | "sep", "rtm", "tsc_adjust", "vme", |
| | "dtes64", "invpcid", "msr", "sse", |
| | "xsave", "smap", "vmx", "erms", "xtpr", |
| | "cmov", "hle", "smep", "nx", "est", |
| | "pat", "monitor", "smx", "pbe", "lm", |
| | "tsc", "adx", "3dnowprefetch", "fpu", |
| | "syscall", "tm", "sse4.1", "fxsr", |
| | "sse4.2", "pclmuldq", "pcid", "pae", |
| | "pni", "tsc-deadline", "mmx", "osxsave", |
| | "cx8", "mce", "de", "tm2", "ht", "dca", |
| | "lahf_lm", "abm", "rdseed", "popcnt", |
| | "mca", "pdpe1gb", "apic", "fsgsbase", |
| | "f16c", "pse", "ds", "invtsc", |
| | "mbm_total", "aes", "avx2", "sse2", |
| | "ss", "ds_cpl", "arat", "bmi1", "bmi2", |
| | "acpi", "ssse3", "rdtscp", "mbm_local", |
| | "cmt", "cx16", "pse36", "mtrr", "movbe", |
| | "pdcm", "fma", "rdrand", "x2apic"] |
| cpu_info_model | Broadwell |
| cpu_info_topology_cells | 2 |
| cpu_info_topology_cores | 22 |
| cpu_info_topology_sockets | 1 |
| cpu_info_topology_threads | 2 |
| cpu_info_vendor | Intel |
| current_workload | 0 |
| disk_available_least | 2 |
| free_disk_gb | 271 |
| free_ram_mb | 49461 |
| host_ip | xxxxxx |
| hypervisor_hostname | xxxxxxx |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 1 |
| local_gb | 394 |
| local_gb_used | 184 |
| memory_mb | 64309 |
| memory_mb_used | 27552 |
| running_vms | 4 |
| service_disabled_reason | None |
| service_host | xxxxxx |
| service_id | 11 |
| state | up |
| status | enabled |
| vcpus | 88 |
| vcpus_used | 8 |
+---------------------------+------------------------------------------+ |
|
| 2017-10-26 12:15:20 |
Khawar Munir Abbasi |
description |
Hi,
I upgraded OpenStack Kolla Newton to Ocata recently. Everything works fine except creating new VMs with small.m1 flavors or larger than this. I can one more cirros VM with tiny.m1 flavor. This means its most likely the problem of space. I checked the logs of neutron,nova and glance. Only nova-conductor logs show error. But that is too generic "No valid host was found". I've investigated that if i somehow increase value of "disk_available_least" in hypervisor then i will create new VMs. Here you can see the relevant details:
nova conductor logs
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to schedule instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 868, in schedule_and_build_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 599, in _schedule_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=self.retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise result
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise exception.NoValidHost(reason=reason)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.486 19 WARNING nova.scheduler.utils [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to compute_task_build_instances: No valid host was found. $
Traceback (most recent call last):
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
return func(*args, **kwargs)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0b6d0bae68d kolla/ubuntu-source-horizon:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks horizon
073f62b32fd1 kolla/ubuntu-source-tacker:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks tacker
af07c296fba1 kolla/ubuntu-source-cron:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks cron
e81b3b76283a kolla/ubuntu-source-mongodb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mongodb
bb0217551442 kolla/ubuntu-source-heat-engine:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_engine
bedfaa812c7a kolla/ubuntu-source-heat-api-cfn:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks heat_api_cfn
669316195ccb kolla/ubuntu-source-heat-api:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_api
b4071a7ed7a7 kolla/ubuntu-source-neutron-metadata-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_metadata_agent
d1cc9cce028e kolla/ubuntu-source-neutron-l3-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_l3_agent
f0aa10f68d96 kolla/ubuntu-source-neutron-dhcp-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_dhcp_agent
6e0a605e7326 kolla/ubuntu-source-neutron-openvswitch-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_openvswitch_agent
f0d4060b71c3 kolla/ubuntu-source-neutron-server:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_server
54a85670ee1b kolla/ubuntu-source-openvswitch-vswitchd:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks openvswitch_vswitchd
13e0f8820afe kolla/ubuntu-source-openvswitch-db-server:4.0.3 "kolla_start" 2 weeks ago Up 17 hours openvswitch_db
fb59adc87585 kolla/ubuntu-source-nova-compute:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_compute
dd76f09e8b41 kolla/ubuntu-source-nova-novncproxy:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_novncproxy
dbb563f6539b kolla/ubuntu-source-nova-consoleauth:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_consoleauth
a2b05c11fe53 kolla/ubuntu-source-nova-conductor:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_conductor
d4783aaff3ee kolla/ubuntu-source-nova-scheduler:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_scheduler
9e1b532d2fcf kolla/ubuntu-source-nova-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_api
08056288cc13 kolla/ubuntu-source-nova-placement-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours placement_api
71a75282a7e6 kolla/ubuntu-source-nova-libvirt:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_libvirt
91788c4b6e18 kolla/ubuntu-source-nova-ssh:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_ssh
4b3e75902d51 kolla/ubuntu-source-glance-registry:4.0.3 "kolla_start" 2 weeks ago Up 17 hours glance_registry
c88023bebdb4 kolla/ubuntu-source-glance-api:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks glance_api
ea150c5b1353 kolla/ubuntu-source-keystone:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks keystone
ad1633c5c3ac kolla/ubuntu-source-rabbitmq:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks rabbitmq
58afd39b647a kolla/ubuntu-source-mariadb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mariadb
9b084133b77a kolla/ubuntu-source-kolla-toolbox:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks kolla_toolbox
2fb94fc409a8 kolla/ubuntu-source-fluentd:4.0.3 "kolla_start" 2 weeks ago Up 17 hours fluentd
8504e5e0879f kolla/ubuntu-source-memcached:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks memcached
nova hypervisor-show xxxxx
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
| cpu_info_arch | x86_64 |
| cpu_info_features | ["pge", "avx", "xsaveopt", "clflush", |
| | "sep", "rtm", "tsc_adjust", "vme", |
| | "dtes64", "invpcid", "msr", "sse", |
| | "xsave", "smap", "vmx", "erms", "xtpr", |
| | "cmov", "hle", "smep", "nx", "est", |
| | "pat", "monitor", "smx", "pbe", "lm", |
| | "tsc", "adx", "3dnowprefetch", "fpu", |
| | "syscall", "tm", "sse4.1", "fxsr", |
| | "sse4.2", "pclmuldq", "pcid", "pae", |
| | "pni", "tsc-deadline", "mmx", "osxsave", |
| | "cx8", "mce", "de", "tm2", "ht", "dca", |
| | "lahf_lm", "abm", "rdseed", "popcnt", |
| | "mca", "pdpe1gb", "apic", "fsgsbase", |
| | "f16c", "pse", "ds", "invtsc", |
| | "mbm_total", "aes", "avx2", "sse2", |
| | "ss", "ds_cpl", "arat", "bmi1", "bmi2", |
| | "acpi", "ssse3", "rdtscp", "mbm_local", |
| | "cmt", "cx16", "pse36", "mtrr", "movbe", |
| | "pdcm", "fma", "rdrand", "x2apic"] |
| cpu_info_model | Broadwell |
| cpu_info_topology_cells | 2 |
| cpu_info_topology_cores | 22 |
| cpu_info_topology_sockets | 1 |
| cpu_info_topology_threads | 2 |
| cpu_info_vendor | Intel |
| current_workload | 0 |
| disk_available_least | 2 |
| free_disk_gb | 271 |
| free_ram_mb | 49461 |
| host_ip | xxxxxx |
| hypervisor_hostname | xxxxxxx |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 1 |
| local_gb | 394 |
| local_gb_used | 184 |
| memory_mb | 64309 |
| memory_mb_used | 27552 |
| running_vms | 4 |
| service_disabled_reason | None |
| service_host | xxxxxx |
| service_id | 11 |
| state | up |
| status | enabled |
| vcpus | 88 |
| vcpus_used | 8 |
+---------------------------+------------------------------------------+ |
Hi,
I upgraded OpenStack Kolla Newton to Ocata recently. Everything works fine except creating new VMs with small.m1 flavors or larger than this. However, I can instantiate one more cirros VM with tiny.m1 flavor. This means its most likely the problem of space. I checked the logs of neutron,nova and glance. Only nova-conductor logs show error. But that is too generic "No valid host was found". I've investigated that if i somehow increase value of "disk_available_least" in hypervisor then i will create new VMs. Here you can see the relevant details:
nova conductor logs
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to schedule instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 868, in schedule_and_build_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 599, in _schedule_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=self.retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise result
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise exception.NoValidHost(reason=reason)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.486 19 WARNING nova.scheduler.utils [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to compute_task_build_instances: No valid host was found. $
Traceback (most recent call last):
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
return func(*args, **kwargs)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0b6d0bae68d kolla/ubuntu-source-horizon:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks horizon
073f62b32fd1 kolla/ubuntu-source-tacker:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks tacker
af07c296fba1 kolla/ubuntu-source-cron:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks cron
e81b3b76283a kolla/ubuntu-source-mongodb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mongodb
bb0217551442 kolla/ubuntu-source-heat-engine:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_engine
bedfaa812c7a kolla/ubuntu-source-heat-api-cfn:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks heat_api_cfn
669316195ccb kolla/ubuntu-source-heat-api:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_api
b4071a7ed7a7 kolla/ubuntu-source-neutron-metadata-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_metadata_agent
d1cc9cce028e kolla/ubuntu-source-neutron-l3-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_l3_agent
f0aa10f68d96 kolla/ubuntu-source-neutron-dhcp-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_dhcp_agent
6e0a605e7326 kolla/ubuntu-source-neutron-openvswitch-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_openvswitch_agent
f0d4060b71c3 kolla/ubuntu-source-neutron-server:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_server
54a85670ee1b kolla/ubuntu-source-openvswitch-vswitchd:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks openvswitch_vswitchd
13e0f8820afe kolla/ubuntu-source-openvswitch-db-server:4.0.3 "kolla_start" 2 weeks ago Up 17 hours openvswitch_db
fb59adc87585 kolla/ubuntu-source-nova-compute:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_compute
dd76f09e8b41 kolla/ubuntu-source-nova-novncproxy:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_novncproxy
dbb563f6539b kolla/ubuntu-source-nova-consoleauth:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_consoleauth
a2b05c11fe53 kolla/ubuntu-source-nova-conductor:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_conductor
d4783aaff3ee kolla/ubuntu-source-nova-scheduler:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_scheduler
9e1b532d2fcf kolla/ubuntu-source-nova-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_api
08056288cc13 kolla/ubuntu-source-nova-placement-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours placement_api
71a75282a7e6 kolla/ubuntu-source-nova-libvirt:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_libvirt
91788c4b6e18 kolla/ubuntu-source-nova-ssh:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_ssh
4b3e75902d51 kolla/ubuntu-source-glance-registry:4.0.3 "kolla_start" 2 weeks ago Up 17 hours glance_registry
c88023bebdb4 kolla/ubuntu-source-glance-api:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks glance_api
ea150c5b1353 kolla/ubuntu-source-keystone:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks keystone
ad1633c5c3ac kolla/ubuntu-source-rabbitmq:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks rabbitmq
58afd39b647a kolla/ubuntu-source-mariadb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mariadb
9b084133b77a kolla/ubuntu-source-kolla-toolbox:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks kolla_toolbox
2fb94fc409a8 kolla/ubuntu-source-fluentd:4.0.3 "kolla_start" 2 weeks ago Up 17 hours fluentd
8504e5e0879f kolla/ubuntu-source-memcached:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks memcached
nova hypervisor-show xxxxx
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
| cpu_info_arch | x86_64 |
| cpu_info_features | ["pge", "avx", "xsaveopt", "clflush", |
| | "sep", "rtm", "tsc_adjust", "vme", |
| | "dtes64", "invpcid", "msr", "sse", |
| | "xsave", "smap", "vmx", "erms", "xtpr", |
| | "cmov", "hle", "smep", "nx", "est", |
| | "pat", "monitor", "smx", "pbe", "lm", |
| | "tsc", "adx", "3dnowprefetch", "fpu", |
| | "syscall", "tm", "sse4.1", "fxsr", |
| | "sse4.2", "pclmuldq", "pcid", "pae", |
| | "pni", "tsc-deadline", "mmx", "osxsave", |
| | "cx8", "mce", "de", "tm2", "ht", "dca", |
| | "lahf_lm", "abm", "rdseed", "popcnt", |
| | "mca", "pdpe1gb", "apic", "fsgsbase", |
| | "f16c", "pse", "ds", "invtsc", |
| | "mbm_total", "aes", "avx2", "sse2", |
| | "ss", "ds_cpl", "arat", "bmi1", "bmi2", |
| | "acpi", "ssse3", "rdtscp", "mbm_local", |
| | "cmt", "cx16", "pse36", "mtrr", "movbe", |
| | "pdcm", "fma", "rdrand", "x2apic"] |
| cpu_info_model | Broadwell |
| cpu_info_topology_cells | 2 |
| cpu_info_topology_cores | 22 |
| cpu_info_topology_sockets | 1 |
| cpu_info_topology_threads | 2 |
| cpu_info_vendor | Intel |
| current_workload | 0 |
| disk_available_least | 2 |
| free_disk_gb | 271 |
| free_ram_mb | 49461 |
| host_ip | xxxxxx |
| hypervisor_hostname | xxxxxxx |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 1 |
| local_gb | 394 |
| local_gb_used | 184 |
| memory_mb | 64309 |
| memory_mb_used | 27552 |
| running_vms | 4 |
| service_disabled_reason | None |
| service_host | xxxxxx |
| service_id | 11 |
| state | up |
| status | enabled |
| vcpus | 88 |
| vcpus_used | 8 |
+---------------------------+------------------------------------------+ |
|
| 2017-10-26 12:15:36 |
Khawar Munir Abbasi |
summary |
in ocata doesn't spin up new VM even having enough resources |
ocata doesn't spin up new VM even having enough resources |
|
| 2017-10-26 12:16:25 |
Khawar Munir Abbasi |
description |
Hi,
I upgraded OpenStack Kolla Newton to Ocata recently. Everything works fine except creating new VMs with small.m1 flavors or larger than this. However, I can instantiate one more cirros VM with tiny.m1 flavor. This means its most likely the problem of space. I checked the logs of neutron,nova and glance. Only nova-conductor logs show error. But that is too generic "No valid host was found". I've investigated that if i somehow increase value of "disk_available_least" in hypervisor then i will create new VMs. Here you can see the relevant details:
nova conductor logs
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to schedule instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 868, in schedule_and_build_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 599, in _schedule_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=self.retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise result
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise exception.NoValidHost(reason=reason)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.486 19 WARNING nova.scheduler.utils [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to compute_task_build_instances: No valid host was found. $
Traceback (most recent call last):
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
return func(*args, **kwargs)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0b6d0bae68d kolla/ubuntu-source-horizon:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks horizon
073f62b32fd1 kolla/ubuntu-source-tacker:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks tacker
af07c296fba1 kolla/ubuntu-source-cron:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks cron
e81b3b76283a kolla/ubuntu-source-mongodb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mongodb
bb0217551442 kolla/ubuntu-source-heat-engine:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_engine
bedfaa812c7a kolla/ubuntu-source-heat-api-cfn:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks heat_api_cfn
669316195ccb kolla/ubuntu-source-heat-api:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_api
b4071a7ed7a7 kolla/ubuntu-source-neutron-metadata-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_metadata_agent
d1cc9cce028e kolla/ubuntu-source-neutron-l3-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_l3_agent
f0aa10f68d96 kolla/ubuntu-source-neutron-dhcp-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_dhcp_agent
6e0a605e7326 kolla/ubuntu-source-neutron-openvswitch-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_openvswitch_agent
f0d4060b71c3 kolla/ubuntu-source-neutron-server:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_server
54a85670ee1b kolla/ubuntu-source-openvswitch-vswitchd:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks openvswitch_vswitchd
13e0f8820afe kolla/ubuntu-source-openvswitch-db-server:4.0.3 "kolla_start" 2 weeks ago Up 17 hours openvswitch_db
fb59adc87585 kolla/ubuntu-source-nova-compute:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_compute
dd76f09e8b41 kolla/ubuntu-source-nova-novncproxy:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_novncproxy
dbb563f6539b kolla/ubuntu-source-nova-consoleauth:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_consoleauth
a2b05c11fe53 kolla/ubuntu-source-nova-conductor:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_conductor
d4783aaff3ee kolla/ubuntu-source-nova-scheduler:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_scheduler
9e1b532d2fcf kolla/ubuntu-source-nova-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_api
08056288cc13 kolla/ubuntu-source-nova-placement-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours placement_api
71a75282a7e6 kolla/ubuntu-source-nova-libvirt:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_libvirt
91788c4b6e18 kolla/ubuntu-source-nova-ssh:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_ssh
4b3e75902d51 kolla/ubuntu-source-glance-registry:4.0.3 "kolla_start" 2 weeks ago Up 17 hours glance_registry
c88023bebdb4 kolla/ubuntu-source-glance-api:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks glance_api
ea150c5b1353 kolla/ubuntu-source-keystone:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks keystone
ad1633c5c3ac kolla/ubuntu-source-rabbitmq:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks rabbitmq
58afd39b647a kolla/ubuntu-source-mariadb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mariadb
9b084133b77a kolla/ubuntu-source-kolla-toolbox:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks kolla_toolbox
2fb94fc409a8 kolla/ubuntu-source-fluentd:4.0.3 "kolla_start" 2 weeks ago Up 17 hours fluentd
8504e5e0879f kolla/ubuntu-source-memcached:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks memcached
nova hypervisor-show xxxxx
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
| cpu_info_arch | x86_64 |
| cpu_info_features | ["pge", "avx", "xsaveopt", "clflush", |
| | "sep", "rtm", "tsc_adjust", "vme", |
| | "dtes64", "invpcid", "msr", "sse", |
| | "xsave", "smap", "vmx", "erms", "xtpr", |
| | "cmov", "hle", "smep", "nx", "est", |
| | "pat", "monitor", "smx", "pbe", "lm", |
| | "tsc", "adx", "3dnowprefetch", "fpu", |
| | "syscall", "tm", "sse4.1", "fxsr", |
| | "sse4.2", "pclmuldq", "pcid", "pae", |
| | "pni", "tsc-deadline", "mmx", "osxsave", |
| | "cx8", "mce", "de", "tm2", "ht", "dca", |
| | "lahf_lm", "abm", "rdseed", "popcnt", |
| | "mca", "pdpe1gb", "apic", "fsgsbase", |
| | "f16c", "pse", "ds", "invtsc", |
| | "mbm_total", "aes", "avx2", "sse2", |
| | "ss", "ds_cpl", "arat", "bmi1", "bmi2", |
| | "acpi", "ssse3", "rdtscp", "mbm_local", |
| | "cmt", "cx16", "pse36", "mtrr", "movbe", |
| | "pdcm", "fma", "rdrand", "x2apic"] |
| cpu_info_model | Broadwell |
| cpu_info_topology_cells | 2 |
| cpu_info_topology_cores | 22 |
| cpu_info_topology_sockets | 1 |
| cpu_info_topology_threads | 2 |
| cpu_info_vendor | Intel |
| current_workload | 0 |
| disk_available_least | 2 |
| free_disk_gb | 271 |
| free_ram_mb | 49461 |
| host_ip | xxxxxx |
| hypervisor_hostname | xxxxxxx |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 1 |
| local_gb | 394 |
| local_gb_used | 184 |
| memory_mb | 64309 |
| memory_mb_used | 27552 |
| running_vms | 4 |
| service_disabled_reason | None |
| service_host | xxxxxx |
| service_id | 11 |
| state | up |
| status | enabled |
| vcpus | 88 |
| vcpus_used | 8 |
+---------------------------+------------------------------------------+ |
Hi,
I upgraded OpenStack Kolla Newton to Ocata recently. Everything works fine except creating new VMs with small.m1 flavors or larger than this. However, I can instantiate one more cirros VM with tiny.m1 flavor. This means its most likely the problem of space. I checked the logs of neutron,nova and glance. Only nova-conductor logs show error. But that is too generic "No valid host was found". I've investigated that if i somehow increase value of "disk_available_least" in hypervisor then i will create new VMs. Here you can see the relevant details:
df -h | more
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 92M 6.2G 2% /run
/dev/mapper/silpixa00397258--vg-root 395G 185G 190G 50% /
tmpfs 32G 11M 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda1 472M 106M 342M 24% /boot
tmpfs 6.3G 4.0K 6.3G 1% /run/user/0
shm 64M 0 64M 0% /var/lib/docker/containers/8504e5e0879f3e31ff981575c9ca4860868bd7ecd8d2d12024ec850d95c46a1c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/9b084133b77a9b2f91ae24a29087b04d575d8a23687803019586fd8c60b70226/shm
shm 64M 0 64M 0% /var/lib/docker/containers/58afd39b647a086f557bedb5a9acad41c5921032a3f18c07f8b5d45451e95673/shm
shm 64M 0 64M 0% /var/lib/docker/containers/ad1633c5c3ac912cd0639c1af76fea0f1ca1bf3db18726777041fe7fdd2e213e/shm
shm 64M 0 64M 0% /var/lib/docker/containers/ea150c5b13533170088b34ae4d11fd1bb21dc8019b901a5b82db291eb1796642/shm
shm 64M 0 64M 0% /var/lib/docker/containers/c88023bebdb4f37ca606441a26261e639ac61ba355f7c7184c357b211321fa38/shm
shm 64M 0 64M 0% /var/lib/docker/containers/54a85670ee1b0e0de977ea72f786f7717d5582354e5795294cb04dcc71988311/shm
shm 64M 0 64M 0% /var/lib/docker/containers/f0d4060b71c3697a94713025d62dcd7a432d885be64b5c3f95def3379ead277f/shm
shm 64M 0 64M 0% /var/lib/docker/containers/6e0a605e73262b775e8caac0541eb70248e5dcab6e4323329f1cc6c636220f5f/shm
shm 64M 0 64M 0% /var/lib/docker/containers/f0aa10f68d96706b81e13a112615d373d97fc12a564a110f24d797115e5928a6/shm
shm 64M 0 64M 0% /var/lib/docker/containers/d1cc9cce028ef50f411b16430a78635d5ae79589a2b31416926d4314102e6f6c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/b4071a7ed7a7c659ec98c2f9edd38a2953ef24d213ff3a6e2e1046df0e61a16c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/bedfaa812c7aa972038c8045edc4c73451d3db1e3633ca3a1d0d9f26034b4d27/shm
shm 64M 0 64M 0% /var/lib/docker/containers/e81b3b76283a683eedd5a3ec7e7add4856a2c0976f7c5b6a6f874419e09fce41/shm
shm 64M 0 64M 0% /var/lib/docker/containers/af07c296fba1223194ed232656c42846c9f2dba478c7cb06ccbeb3e5ec0ba828/shm
shm 64M 0 64M 0% /var/lib/docker/containers/073f62b32fd175552adaa24298a6f8529f0afbf14fd7024da5cda69f72d9a028/shm
shm 64M 0 64M 0% /var/lib/docker/containers/c0b6d0bae68da337500cbe04f35e8142c067ae918e1bcd33e9afa7d866279f08/shm
cgmfs 100K 0 100K 0% /run/cgmanager/fs
shm 64M 0 64M 0% /var/lib/docker/containers/bb0217551442a405d82b89a4da66030b412596520c08e8643cafb4ab57fb7f35/shm
shm 64M 0 64M 0% /var/lib/docker/containers/669316195ccbb17470630b179438fe0755f31d52afbc836294bb02a9fa342e14/shm
shm 64M 0 64M 0% /var/lib/docker/containers/13e0f8820afeea16c9bee5dfc5a5316a6baf47206671bc5a74df03655acb464f/shm
shm 64M 0 64M 0% /var/lib/docker/containers/dd76f09e8b41541a01894db81413e336b6f86f25e6d14266ee3e7f94677fe811/shm
shm 64M 0 64M 0% /var/lib/docker/containers/dbb563f6539bc21404dee8a98e6a9a05ea4bba02dbdc0c5ea9c0e82039794027/shm
shm 64M 0 64M 0% /var/lib/docker/containers/a2b05c11fe53060bbad47ce4e612317ee8c86117a8ee5242ef3eecd24687b31a/shm
shm 64M 0 64M 0% /var/lib/docker/containers/9e1b532d2fcfe206621176f938a72622b2d361cc582e3b0c518ec7a52dd87f31/shm
shm 64M 0 64M 0% /var/lib/docker/containers/08056288cc13db2d110b1bae76628d3f64522eea6a70d1592766df2832d81570/shm
shm 64M 0 64M 0% /var/lib/docker/containers/4b3e75902d5139328173178dbbcdd92502f2e72693224d0b2139072bee3b6e3c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/2fb94fc409a845266cd3a670b9384943f823062e5e1f0161ed131f44b4ed4231/shm
shm 64M 0 64M 0% /var/lib/docker/containers/91788c4b6e18e841271390beee04660d8627a20313fc4d26d37b15025dcc3a7e/shm
shm 64M 0 64M 0% /var/lib/docker/containers/fb59adc87585756916cefaa863e58652d4a2c5bab8b1cf0bc9ffd93ca2e8466a/shm
shm 64M 0 64M 0% /var/lib/docker/containers/71a75282a7e6226133469b5f5894c23942e400654aabb9085a59252d59614533/shm
shm 64M 0 64M 0% /var/lib/docker/containers/d4783aaff3ee5e3ca51202548435640248a87ec615d54360ecfee9cb072b9168/shm
nova conductor logs
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to schedule instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 868, in schedule_and_build_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 599, in _schedule_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=self.retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise result
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise exception.NoValidHost(reason=reason)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.486 19 WARNING nova.scheduler.utils [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to compute_task_build_instances: No valid host was found. $
Traceback (most recent call last):
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
return func(*args, **kwargs)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0b6d0bae68d kolla/ubuntu-source-horizon:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks horizon
073f62b32fd1 kolla/ubuntu-source-tacker:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks tacker
af07c296fba1 kolla/ubuntu-source-cron:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks cron
e81b3b76283a kolla/ubuntu-source-mongodb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mongodb
bb0217551442 kolla/ubuntu-source-heat-engine:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_engine
bedfaa812c7a kolla/ubuntu-source-heat-api-cfn:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks heat_api_cfn
669316195ccb kolla/ubuntu-source-heat-api:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_api
b4071a7ed7a7 kolla/ubuntu-source-neutron-metadata-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_metadata_agent
d1cc9cce028e kolla/ubuntu-source-neutron-l3-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_l3_agent
f0aa10f68d96 kolla/ubuntu-source-neutron-dhcp-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_dhcp_agent
6e0a605e7326 kolla/ubuntu-source-neutron-openvswitch-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_openvswitch_agent
f0d4060b71c3 kolla/ubuntu-source-neutron-server:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_server
54a85670ee1b kolla/ubuntu-source-openvswitch-vswitchd:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks openvswitch_vswitchd
13e0f8820afe kolla/ubuntu-source-openvswitch-db-server:4.0.3 "kolla_start" 2 weeks ago Up 17 hours openvswitch_db
fb59adc87585 kolla/ubuntu-source-nova-compute:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_compute
dd76f09e8b41 kolla/ubuntu-source-nova-novncproxy:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_novncproxy
dbb563f6539b kolla/ubuntu-source-nova-consoleauth:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_consoleauth
a2b05c11fe53 kolla/ubuntu-source-nova-conductor:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_conductor
d4783aaff3ee kolla/ubuntu-source-nova-scheduler:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_scheduler
9e1b532d2fcf kolla/ubuntu-source-nova-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_api
08056288cc13 kolla/ubuntu-source-nova-placement-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours placement_api
71a75282a7e6 kolla/ubuntu-source-nova-libvirt:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_libvirt
91788c4b6e18 kolla/ubuntu-source-nova-ssh:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_ssh
4b3e75902d51 kolla/ubuntu-source-glance-registry:4.0.3 "kolla_start" 2 weeks ago Up 17 hours glance_registry
c88023bebdb4 kolla/ubuntu-source-glance-api:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks glance_api
ea150c5b1353 kolla/ubuntu-source-keystone:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks keystone
ad1633c5c3ac kolla/ubuntu-source-rabbitmq:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks rabbitmq
58afd39b647a kolla/ubuntu-source-mariadb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mariadb
9b084133b77a kolla/ubuntu-source-kolla-toolbox:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks kolla_toolbox
2fb94fc409a8 kolla/ubuntu-source-fluentd:4.0.3 "kolla_start" 2 weeks ago Up 17 hours fluentd
8504e5e0879f kolla/ubuntu-source-memcached:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks memcached
nova hypervisor-show xxxxx
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
| cpu_info_arch | x86_64 |
| cpu_info_features | ["pge", "avx", "xsaveopt", "clflush", |
| | "sep", "rtm", "tsc_adjust", "vme", |
| | "dtes64", "invpcid", "msr", "sse", |
| | "xsave", "smap", "vmx", "erms", "xtpr", |
| | "cmov", "hle", "smep", "nx", "est", |
| | "pat", "monitor", "smx", "pbe", "lm", |
| | "tsc", "adx", "3dnowprefetch", "fpu", |
| | "syscall", "tm", "sse4.1", "fxsr", |
| | "sse4.2", "pclmuldq", "pcid", "pae", |
| | "pni", "tsc-deadline", "mmx", "osxsave", |
| | "cx8", "mce", "de", "tm2", "ht", "dca", |
| | "lahf_lm", "abm", "rdseed", "popcnt", |
| | "mca", "pdpe1gb", "apic", "fsgsbase", |
| | "f16c", "pse", "ds", "invtsc", |
| | "mbm_total", "aes", "avx2", "sse2", |
| | "ss", "ds_cpl", "arat", "bmi1", "bmi2", |
| | "acpi", "ssse3", "rdtscp", "mbm_local", |
| | "cmt", "cx16", "pse36", "mtrr", "movbe", |
| | "pdcm", "fma", "rdrand", "x2apic"] |
| cpu_info_model | Broadwell |
| cpu_info_topology_cells | 2 |
| cpu_info_topology_cores | 22 |
| cpu_info_topology_sockets | 1 |
| cpu_info_topology_threads | 2 |
| cpu_info_vendor | Intel |
| current_workload | 0 |
| disk_available_least | 2 |
| free_disk_gb | 271 |
| free_ram_mb | 49461 |
| host_ip | xxxxxx |
| hypervisor_hostname | xxxxxxx |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 1 |
| local_gb | 394 |
| local_gb_used | 184 |
| memory_mb | 64309 |
| memory_mb_used | 27552 |
| running_vms | 4 |
| service_disabled_reason | None |
| service_host | xxxxxx |
| service_id | 11 |
| state | up |
| status | enabled |
| vcpus | 88 |
| vcpus_used | 8 |
+---------------------------+------------------------------------------+ |
|
| 2017-10-26 12:17:25 |
Khawar Munir Abbasi |
description |
Hi,
I upgraded OpenStack Kolla Newton to Ocata recently. Everything works fine except creating new VMs with small.m1 flavors or larger than this. However, I can instantiate one more cirros VM with tiny.m1 flavor. This means its most likely the problem of space. I checked the logs of neutron,nova and glance. Only nova-conductor logs show error. But that is too generic "No valid host was found". I've investigated that if i somehow increase value of "disk_available_least" in hypervisor then i will create new VMs. Here you can see the relevant details:
df -h | more
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 92M 6.2G 2% /run
/dev/mapper/silpixa00397258--vg-root 395G 185G 190G 50% /
tmpfs 32G 11M 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda1 472M 106M 342M 24% /boot
tmpfs 6.3G 4.0K 6.3G 1% /run/user/0
shm 64M 0 64M 0% /var/lib/docker/containers/8504e5e0879f3e31ff981575c9ca4860868bd7ecd8d2d12024ec850d95c46a1c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/9b084133b77a9b2f91ae24a29087b04d575d8a23687803019586fd8c60b70226/shm
shm 64M 0 64M 0% /var/lib/docker/containers/58afd39b647a086f557bedb5a9acad41c5921032a3f18c07f8b5d45451e95673/shm
shm 64M 0 64M 0% /var/lib/docker/containers/ad1633c5c3ac912cd0639c1af76fea0f1ca1bf3db18726777041fe7fdd2e213e/shm
shm 64M 0 64M 0% /var/lib/docker/containers/ea150c5b13533170088b34ae4d11fd1bb21dc8019b901a5b82db291eb1796642/shm
shm 64M 0 64M 0% /var/lib/docker/containers/c88023bebdb4f37ca606441a26261e639ac61ba355f7c7184c357b211321fa38/shm
shm 64M 0 64M 0% /var/lib/docker/containers/54a85670ee1b0e0de977ea72f786f7717d5582354e5795294cb04dcc71988311/shm
shm 64M 0 64M 0% /var/lib/docker/containers/f0d4060b71c3697a94713025d62dcd7a432d885be64b5c3f95def3379ead277f/shm
shm 64M 0 64M 0% /var/lib/docker/containers/6e0a605e73262b775e8caac0541eb70248e5dcab6e4323329f1cc6c636220f5f/shm
shm 64M 0 64M 0% /var/lib/docker/containers/f0aa10f68d96706b81e13a112615d373d97fc12a564a110f24d797115e5928a6/shm
shm 64M 0 64M 0% /var/lib/docker/containers/d1cc9cce028ef50f411b16430a78635d5ae79589a2b31416926d4314102e6f6c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/b4071a7ed7a7c659ec98c2f9edd38a2953ef24d213ff3a6e2e1046df0e61a16c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/bedfaa812c7aa972038c8045edc4c73451d3db1e3633ca3a1d0d9f26034b4d27/shm
shm 64M 0 64M 0% /var/lib/docker/containers/e81b3b76283a683eedd5a3ec7e7add4856a2c0976f7c5b6a6f874419e09fce41/shm
shm 64M 0 64M 0% /var/lib/docker/containers/af07c296fba1223194ed232656c42846c9f2dba478c7cb06ccbeb3e5ec0ba828/shm
shm 64M 0 64M 0% /var/lib/docker/containers/073f62b32fd175552adaa24298a6f8529f0afbf14fd7024da5cda69f72d9a028/shm
shm 64M 0 64M 0% /var/lib/docker/containers/c0b6d0bae68da337500cbe04f35e8142c067ae918e1bcd33e9afa7d866279f08/shm
cgmfs 100K 0 100K 0% /run/cgmanager/fs
shm 64M 0 64M 0% /var/lib/docker/containers/bb0217551442a405d82b89a4da66030b412596520c08e8643cafb4ab57fb7f35/shm
shm 64M 0 64M 0% /var/lib/docker/containers/669316195ccbb17470630b179438fe0755f31d52afbc836294bb02a9fa342e14/shm
shm 64M 0 64M 0% /var/lib/docker/containers/13e0f8820afeea16c9bee5dfc5a5316a6baf47206671bc5a74df03655acb464f/shm
shm 64M 0 64M 0% /var/lib/docker/containers/dd76f09e8b41541a01894db81413e336b6f86f25e6d14266ee3e7f94677fe811/shm
shm 64M 0 64M 0% /var/lib/docker/containers/dbb563f6539bc21404dee8a98e6a9a05ea4bba02dbdc0c5ea9c0e82039794027/shm
shm 64M 0 64M 0% /var/lib/docker/containers/a2b05c11fe53060bbad47ce4e612317ee8c86117a8ee5242ef3eecd24687b31a/shm
shm 64M 0 64M 0% /var/lib/docker/containers/9e1b532d2fcfe206621176f938a72622b2d361cc582e3b0c518ec7a52dd87f31/shm
shm 64M 0 64M 0% /var/lib/docker/containers/08056288cc13db2d110b1bae76628d3f64522eea6a70d1592766df2832d81570/shm
shm 64M 0 64M 0% /var/lib/docker/containers/4b3e75902d5139328173178dbbcdd92502f2e72693224d0b2139072bee3b6e3c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/2fb94fc409a845266cd3a670b9384943f823062e5e1f0161ed131f44b4ed4231/shm
shm 64M 0 64M 0% /var/lib/docker/containers/91788c4b6e18e841271390beee04660d8627a20313fc4d26d37b15025dcc3a7e/shm
shm 64M 0 64M 0% /var/lib/docker/containers/fb59adc87585756916cefaa863e58652d4a2c5bab8b1cf0bc9ffd93ca2e8466a/shm
shm 64M 0 64M 0% /var/lib/docker/containers/71a75282a7e6226133469b5f5894c23942e400654aabb9085a59252d59614533/shm
shm 64M 0 64M 0% /var/lib/docker/containers/d4783aaff3ee5e3ca51202548435640248a87ec615d54360ecfee9cb072b9168/shm
nova conductor logs
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to schedule instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 868, in schedule_and_build_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 599, in _schedule_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=self.retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise result
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise exception.NoValidHost(reason=reason)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.486 19 WARNING nova.scheduler.utils [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to compute_task_build_instances: No valid host was found. $
Traceback (most recent call last):
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
return func(*args, **kwargs)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0b6d0bae68d kolla/ubuntu-source-horizon:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks horizon
073f62b32fd1 kolla/ubuntu-source-tacker:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks tacker
af07c296fba1 kolla/ubuntu-source-cron:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks cron
e81b3b76283a kolla/ubuntu-source-mongodb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mongodb
bb0217551442 kolla/ubuntu-source-heat-engine:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_engine
bedfaa812c7a kolla/ubuntu-source-heat-api-cfn:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks heat_api_cfn
669316195ccb kolla/ubuntu-source-heat-api:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_api
b4071a7ed7a7 kolla/ubuntu-source-neutron-metadata-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_metadata_agent
d1cc9cce028e kolla/ubuntu-source-neutron-l3-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_l3_agent
f0aa10f68d96 kolla/ubuntu-source-neutron-dhcp-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_dhcp_agent
6e0a605e7326 kolla/ubuntu-source-neutron-openvswitch-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_openvswitch_agent
f0d4060b71c3 kolla/ubuntu-source-neutron-server:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_server
54a85670ee1b kolla/ubuntu-source-openvswitch-vswitchd:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks openvswitch_vswitchd
13e0f8820afe kolla/ubuntu-source-openvswitch-db-server:4.0.3 "kolla_start" 2 weeks ago Up 17 hours openvswitch_db
fb59adc87585 kolla/ubuntu-source-nova-compute:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_compute
dd76f09e8b41 kolla/ubuntu-source-nova-novncproxy:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_novncproxy
dbb563f6539b kolla/ubuntu-source-nova-consoleauth:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_consoleauth
a2b05c11fe53 kolla/ubuntu-source-nova-conductor:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_conductor
d4783aaff3ee kolla/ubuntu-source-nova-scheduler:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_scheduler
9e1b532d2fcf kolla/ubuntu-source-nova-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_api
08056288cc13 kolla/ubuntu-source-nova-placement-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours placement_api
71a75282a7e6 kolla/ubuntu-source-nova-libvirt:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_libvirt
91788c4b6e18 kolla/ubuntu-source-nova-ssh:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_ssh
4b3e75902d51 kolla/ubuntu-source-glance-registry:4.0.3 "kolla_start" 2 weeks ago Up 17 hours glance_registry
c88023bebdb4 kolla/ubuntu-source-glance-api:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks glance_api
ea150c5b1353 kolla/ubuntu-source-keystone:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks keystone
ad1633c5c3ac kolla/ubuntu-source-rabbitmq:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks rabbitmq
58afd39b647a kolla/ubuntu-source-mariadb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mariadb
9b084133b77a kolla/ubuntu-source-kolla-toolbox:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks kolla_toolbox
2fb94fc409a8 kolla/ubuntu-source-fluentd:4.0.3 "kolla_start" 2 weeks ago Up 17 hours fluentd
8504e5e0879f kolla/ubuntu-source-memcached:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks memcached
nova hypervisor-show xxxxx
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
| cpu_info_arch | x86_64 |
| cpu_info_features | ["pge", "avx", "xsaveopt", "clflush", |
| | "sep", "rtm", "tsc_adjust", "vme", |
| | "dtes64", "invpcid", "msr", "sse", |
| | "xsave", "smap", "vmx", "erms", "xtpr", |
| | "cmov", "hle", "smep", "nx", "est", |
| | "pat", "monitor", "smx", "pbe", "lm", |
| | "tsc", "adx", "3dnowprefetch", "fpu", |
| | "syscall", "tm", "sse4.1", "fxsr", |
| | "sse4.2", "pclmuldq", "pcid", "pae", |
| | "pni", "tsc-deadline", "mmx", "osxsave", |
| | "cx8", "mce", "de", "tm2", "ht", "dca", |
| | "lahf_lm", "abm", "rdseed", "popcnt", |
| | "mca", "pdpe1gb", "apic", "fsgsbase", |
| | "f16c", "pse", "ds", "invtsc", |
| | "mbm_total", "aes", "avx2", "sse2", |
| | "ss", "ds_cpl", "arat", "bmi1", "bmi2", |
| | "acpi", "ssse3", "rdtscp", "mbm_local", |
| | "cmt", "cx16", "pse36", "mtrr", "movbe", |
| | "pdcm", "fma", "rdrand", "x2apic"] |
| cpu_info_model | Broadwell |
| cpu_info_topology_cells | 2 |
| cpu_info_topology_cores | 22 |
| cpu_info_topology_sockets | 1 |
| cpu_info_topology_threads | 2 |
| cpu_info_vendor | Intel |
| current_workload | 0 |
| disk_available_least | 2 |
| free_disk_gb | 271 |
| free_ram_mb | 49461 |
| host_ip | xxxxxx |
| hypervisor_hostname | xxxxxxx |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 1 |
| local_gb | 394 |
| local_gb_used | 184 |
| memory_mb | 64309 |
| memory_mb_used | 27552 |
| running_vms | 4 |
| service_disabled_reason | None |
| service_host | xxxxxx |
| service_id | 11 |
| state | up |
| status | enabled |
| vcpus | 88 |
| vcpus_used | 8 |
+---------------------------+------------------------------------------+ |
Hi,
I upgraded OpenStack Kolla Newton to Ocata recently. Everything works fine except creating new VMs with small.m1 flavors or larger than this. However, I can instantiate one more cirros VM with tiny.m1 flavor. This means its most likely the problem of space. I checked the logs of neutron,nova and glance. Only nova-conductor logs show error. But that is too generic "No valid host was found". I've investigated that if i somehow increase value of "disk_available_least" in hypervisor then i will create new VMs. Here you can see the relevant details:
df -h | more
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 92M 6.2G 2% /run
/dev/mapper/xxx --vg-root 395G 185G 190G 50% /
tmpfs 32G 11M 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda1 472M 106M 342M 24% /boot
tmpfs 6.3G 4.0K 6.3G 1% /run/user/0
shm 64M 0 64M 0% /var/lib/docker/containers/8504e5e0879f3e31ff981575c9ca4860868bd7ecd8d2d12024ec850d95c46a1c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/9b084133b77a9b2f91ae24a29087b04d575d8a23687803019586fd8c60b70226/shm
shm 64M 0 64M 0% /var/lib/docker/containers/58afd39b647a086f557bedb5a9acad41c5921032a3f18c07f8b5d45451e95673/shm
shm 64M 0 64M 0% /var/lib/docker/containers/ad1633c5c3ac912cd0639c1af76fea0f1ca1bf3db18726777041fe7fdd2e213e/shm
shm 64M 0 64M 0% /var/lib/docker/containers/ea150c5b13533170088b34ae4d11fd1bb21dc8019b901a5b82db291eb1796642/shm
shm 64M 0 64M 0% /var/lib/docker/containers/c88023bebdb4f37ca606441a26261e639ac61ba355f7c7184c357b211321fa38/shm
shm 64M 0 64M 0% /var/lib/docker/containers/54a85670ee1b0e0de977ea72f786f7717d5582354e5795294cb04dcc71988311/shm
shm 64M 0 64M 0% /var/lib/docker/containers/f0d4060b71c3697a94713025d62dcd7a432d885be64b5c3f95def3379ead277f/shm
shm 64M 0 64M 0% /var/lib/docker/containers/6e0a605e73262b775e8caac0541eb70248e5dcab6e4323329f1cc6c636220f5f/shm
shm 64M 0 64M 0% /var/lib/docker/containers/f0aa10f68d96706b81e13a112615d373d97fc12a564a110f24d797115e5928a6/shm
shm 64M 0 64M 0% /var/lib/docker/containers/d1cc9cce028ef50f411b16430a78635d5ae79589a2b31416926d4314102e6f6c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/b4071a7ed7a7c659ec98c2f9edd38a2953ef24d213ff3a6e2e1046df0e61a16c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/bedfaa812c7aa972038c8045edc4c73451d3db1e3633ca3a1d0d9f26034b4d27/shm
shm 64M 0 64M 0% /var/lib/docker/containers/e81b3b76283a683eedd5a3ec7e7add4856a2c0976f7c5b6a6f874419e09fce41/shm
shm 64M 0 64M 0% /var/lib/docker/containers/af07c296fba1223194ed232656c42846c9f2dba478c7cb06ccbeb3e5ec0ba828/shm
shm 64M 0 64M 0% /var/lib/docker/containers/073f62b32fd175552adaa24298a6f8529f0afbf14fd7024da5cda69f72d9a028/shm
shm 64M 0 64M 0% /var/lib/docker/containers/c0b6d0bae68da337500cbe04f35e8142c067ae918e1bcd33e9afa7d866279f08/shm
cgmfs 100K 0 100K 0% /run/cgmanager/fs
shm 64M 0 64M 0% /var/lib/docker/containers/bb0217551442a405d82b89a4da66030b412596520c08e8643cafb4ab57fb7f35/shm
shm 64M 0 64M 0% /var/lib/docker/containers/669316195ccbb17470630b179438fe0755f31d52afbc836294bb02a9fa342e14/shm
shm 64M 0 64M 0% /var/lib/docker/containers/13e0f8820afeea16c9bee5dfc5a5316a6baf47206671bc5a74df03655acb464f/shm
shm 64M 0 64M 0% /var/lib/docker/containers/dd76f09e8b41541a01894db81413e336b6f86f25e6d14266ee3e7f94677fe811/shm
shm 64M 0 64M 0% /var/lib/docker/containers/dbb563f6539bc21404dee8a98e6a9a05ea4bba02dbdc0c5ea9c0e82039794027/shm
shm 64M 0 64M 0% /var/lib/docker/containers/a2b05c11fe53060bbad47ce4e612317ee8c86117a8ee5242ef3eecd24687b31a/shm
shm 64M 0 64M 0% /var/lib/docker/containers/9e1b532d2fcfe206621176f938a72622b2d361cc582e3b0c518ec7a52dd87f31/shm
shm 64M 0 64M 0% /var/lib/docker/containers/08056288cc13db2d110b1bae76628d3f64522eea6a70d1592766df2832d81570/shm
shm 64M 0 64M 0% /var/lib/docker/containers/4b3e75902d5139328173178dbbcdd92502f2e72693224d0b2139072bee3b6e3c/shm
shm 64M 0 64M 0% /var/lib/docker/containers/2fb94fc409a845266cd3a670b9384943f823062e5e1f0161ed131f44b4ed4231/shm
shm 64M 0 64M 0% /var/lib/docker/containers/91788c4b6e18e841271390beee04660d8627a20313fc4d26d37b15025dcc3a7e/shm
shm 64M 0 64M 0% /var/lib/docker/containers/fb59adc87585756916cefaa863e58652d4a2c5bab8b1cf0bc9ffd93ca2e8466a/shm
shm 64M 0 64M 0% /var/lib/docker/containers/71a75282a7e6226133469b5f5894c23942e400654aabb9085a59252d59614533/shm
shm 64M 0 64M 0% /var/lib/docker/containers/d4783aaff3ee5e3ca51202548435640248a87ec615d54360ecfee9cb072b9168/shm
nova conductor logs
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to schedule instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 868, in schedule_and_build_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py", line 599, in _schedule_instances
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=self.retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager retry=retry)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 569, in _send
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise result
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager raise exception.NoValidHost(reason=reason)
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.424 19 ERROR nova.conductor.manager
2017-10-26 13:05:31.486 19 WARNING nova.scheduler.utils [req-4ce86580-53a9-4a7a-ab77-7e77b69900a6 5e5fd529653545669d37817e2286298e 1b5dd3946a2d4918afafedf95887b659 - - -] Failed to compute_task_build_instances: No valid host was found. $
Traceback (most recent call last):
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 218, in inner
return func(*args, **kwargs)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/manager.py", line 98, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)
File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0b6d0bae68d kolla/ubuntu-source-horizon:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks horizon
073f62b32fd1 kolla/ubuntu-source-tacker:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks tacker
af07c296fba1 kolla/ubuntu-source-cron:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks cron
e81b3b76283a kolla/ubuntu-source-mongodb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mongodb
bb0217551442 kolla/ubuntu-source-heat-engine:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_engine
bedfaa812c7a kolla/ubuntu-source-heat-api-cfn:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks heat_api_cfn
669316195ccb kolla/ubuntu-source-heat-api:4.0.3 "kolla_start" 2 weeks ago Up 12 days heat_api
b4071a7ed7a7 kolla/ubuntu-source-neutron-metadata-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_metadata_agent
d1cc9cce028e kolla/ubuntu-source-neutron-l3-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_l3_agent
f0aa10f68d96 kolla/ubuntu-source-neutron-dhcp-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_dhcp_agent
6e0a605e7326 kolla/ubuntu-source-neutron-openvswitch-agent:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_openvswitch_agent
f0d4060b71c3 kolla/ubuntu-source-neutron-server:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks neutron_server
54a85670ee1b kolla/ubuntu-source-openvswitch-vswitchd:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks openvswitch_vswitchd
13e0f8820afe kolla/ubuntu-source-openvswitch-db-server:4.0.3 "kolla_start" 2 weeks ago Up 17 hours openvswitch_db
fb59adc87585 kolla/ubuntu-source-nova-compute:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_compute
dd76f09e8b41 kolla/ubuntu-source-nova-novncproxy:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_novncproxy
dbb563f6539b kolla/ubuntu-source-nova-consoleauth:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_consoleauth
a2b05c11fe53 kolla/ubuntu-source-nova-conductor:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_conductor
d4783aaff3ee kolla/ubuntu-source-nova-scheduler:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_scheduler
9e1b532d2fcf kolla/ubuntu-source-nova-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_api
08056288cc13 kolla/ubuntu-source-nova-placement-api:4.0.3 "kolla_start" 2 weeks ago Up 17 hours placement_api
71a75282a7e6 kolla/ubuntu-source-nova-libvirt:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_libvirt
91788c4b6e18 kolla/ubuntu-source-nova-ssh:4.0.3 "kolla_start" 2 weeks ago Up 17 hours nova_ssh
4b3e75902d51 kolla/ubuntu-source-glance-registry:4.0.3 "kolla_start" 2 weeks ago Up 17 hours glance_registry
c88023bebdb4 kolla/ubuntu-source-glance-api:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks glance_api
ea150c5b1353 kolla/ubuntu-source-keystone:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks keystone
ad1633c5c3ac kolla/ubuntu-source-rabbitmq:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks rabbitmq
58afd39b647a kolla/ubuntu-source-mariadb:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks mariadb
9b084133b77a kolla/ubuntu-source-kolla-toolbox:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks kolla_toolbox
2fb94fc409a8 kolla/ubuntu-source-fluentd:4.0.3 "kolla_start" 2 weeks ago Up 17 hours fluentd
8504e5e0879f kolla/ubuntu-source-memcached:4.0.3 "kolla_start" 2 weeks ago Up 2 weeks memcached
nova hypervisor-show xxxxx
+---------------------------+------------------------------------------+
| Property | Value |
+---------------------------+------------------------------------------+
| cpu_info_arch | x86_64 |
| cpu_info_features | ["pge", "avx", "xsaveopt", "clflush", |
| | "sep", "rtm", "tsc_adjust", "vme", |
| | "dtes64", "invpcid", "msr", "sse", |
| | "xsave", "smap", "vmx", "erms", "xtpr", |
| | "cmov", "hle", "smep", "nx", "est", |
| | "pat", "monitor", "smx", "pbe", "lm", |
| | "tsc", "adx", "3dnowprefetch", "fpu", |
| | "syscall", "tm", "sse4.1", "fxsr", |
| | "sse4.2", "pclmuldq", "pcid", "pae", |
| | "pni", "tsc-deadline", "mmx", "osxsave", |
| | "cx8", "mce", "de", "tm2", "ht", "dca", |
| | "lahf_lm", "abm", "rdseed", "popcnt", |
| | "mca", "pdpe1gb", "apic", "fsgsbase", |
| | "f16c", "pse", "ds", "invtsc", |
| | "mbm_total", "aes", "avx2", "sse2", |
| | "ss", "ds_cpl", "arat", "bmi1", "bmi2", |
| | "acpi", "ssse3", "rdtscp", "mbm_local", |
| | "cmt", "cx16", "pse36", "mtrr", "movbe", |
| | "pdcm", "fma", "rdrand", "x2apic"] |
| cpu_info_model | Broadwell |
| cpu_info_topology_cells | 2 |
| cpu_info_topology_cores | 22 |
| cpu_info_topology_sockets | 1 |
| cpu_info_topology_threads | 2 |
| cpu_info_vendor | Intel |
| current_workload | 0 |
| disk_available_least | 2 |
| free_disk_gb | 271 |
| free_ram_mb | 49461 |
| host_ip | xxxxxx |
| hypervisor_hostname | xxxxxxx |
| hypervisor_type | QEMU |
| hypervisor_version | 2008000 |
| id | 1 |
| local_gb | 394 |
| local_gb_used | 184 |
| memory_mb | 64309 |
| memory_mb_used | 27552 |
| running_vms | 4 |
| service_disabled_reason | None |
| service_host | xxxxxx |
| service_id | 11 |
| state | up |
| status | enabled |
| vcpus | 88 |
| vcpus_used | 8 |
+---------------------------+------------------------------------------+ |
|
| 2017-10-26 16:26:59 |
sean mooney |
kolla: status |
New |
Incomplete |
|
| 2017-12-27 04:17:23 |
Launchpad Janitor |
kolla: status |
Incomplete |
Expired |
|
