Nova Fails to Boot VMs

[Daneyon Hansen]

I successfully build images (--base ubuntu and --type source), run kolla-deploy, and run init-runonce. I now try to boot a VM:

root@k03-13:~/kolla# nova boot --image cirros --flavor m1.tiny --key-name mykey --nic net-id=36fd3270-2304-403a-9a95-3fb67941fed7 c1
+--------------------------------------+-----------------------------------------------+
| Property | Value |
+--------------------------------------+-----------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | 59AtR2CippND |
| config_drive | |
| created | 2016-04-22T20:24:23Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 19c79c54-a592-494c-9dd4-9079eb1a0ff4 |
| image | cirros (d41ad962-fbcb-489e-9547-cd0a568cb0ea) |
| key_name | mykey |
| metadata | {} |
| name | c1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 4c08d171fa234361aa0ecf057384e215 |
| updated | 2016-04-22T20:24:23Z |
| user_id | 39b43b3464cc4558845163d3f30a6194 |
+--------------------------------------+-----------------------------------------------+

The VM's error:

root@k03-13:~/kolla# nova list
+--------------------------------------+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+----------+
| 19c79c54-a592-494c-9dd4-9079eb1a0ff4 | c1 | ERROR | - | NOSTATE | |
| 2357f08d-dfe6-4ee4-962c-b92b27eeb934 | c2 | ERROR | - | NOSTATE | |
+--------------------------------------+------+--------+------------+-------------+----------+

With these details:

root@k03-13:~/kolla# nova show c1
+--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-04-22T20:24:23Z |
| fault | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": " File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py\", line 392, in build_instances |
| | context, request_spec, filter_properties) |
| | File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/conductor/manager.py\", line 436, in _schedule_instances |
| | hosts = self.scheduler_client.select_destinations(context, spec_obj) |
| | File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 372, in wrapped |
| | return func(*args, **kwargs) |
| | File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 51, in select_destinations |
| | return self.queryclient.select_destinations(context, spec_obj) |
| | File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 37, in __run_method |
| | return getattr(self.instance, __name)(*args, **kwargs) |
| | File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/client/query.py\", line 32, in select_destinations |
| | return self.scheduler_rpcapi.select_destinations(context, spec_obj) |
| | File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\", line 121, in select_destinations |
| | return cctxt.call(ctxt, 'select_destinations', **msg_args) |
| | File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line 158, in call |
| | retry=self.retry) |
| | File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 90, in _send |
| | timeout=timeout, retry=retry) |
| | File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 470, in send |
| | retry=retry) |
| | File \"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 461, in _send |
| | raise result |
| | ", "created": "2016-04-22T20:24:29Z"} |
| flavor | m1.tiny (1) |
| hostId | |
| id | 19c79c54-a592-494c-9dd4-9079eb1a0ff4 |
| image | cirros (d41ad962-fbcb-489e-9547-cd0a568cb0ea) |
| key_name | mykey |
| metadata | {} |
| name | c1 |
| os-extended-volumes:volumes_attached | [] |
| status | ERROR |
| tenant_id | 4c08d171fa234361aa0ecf057384e215 |
| updated | 2016-04-22T20:24:30Z |
| user_id | 39b43b3464cc4558845163d3f30a6194 |
+--------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Nova services are running:

root@k03-13:~/kolla# nova service-list
+----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+--------+----------+---------+-------+----------------------------+-----------------+
| 2 | nova-consoleauth | k03-13 | internal | enabled | up | 2016-04-22T20:59:55.000000 | - |
| 6 | nova-scheduler | k03-13 | internal | enabled | up | 2016-04-22T20:59:55.000000 | - |
| 7 | nova-conductor | k03-13 | internal | enabled | up | 2016-04-22T20:59:55.000000 | - |
| 8 | nova-compute | k03-13 | nova | enabled | up | 2016-04-22T21:00:00.000000 | - |
+----+------------------+--------+----------+---------+-------+----------------------------+-----------------+

I check hw-virt:
root@k03-13:~/kolla# egrep -c '(vmx|svm)' /proc/cpuinfo
8

Nova config:
root@k03-13:~/kolla# cat /etc/kolla/nova-compute/nova.conf
[DEFAULT]
debug = True
log_dir = /var/log/kolla/nova
use_forwarded_for = true
api_paste_config = /etc/nova/api-paste.ini
state_path = /var/lib/nova
osapi_compute_listen = 10.30.118.91
osapi_compute_listen_port = 8774
metadata_listen = 10.30.118.91
metadata_listen_port = 8775
ec2_listen = 10.30.118.91
ec2_listen_port = 8773
use_neutron = True
firewall_driver = nova.virt.firewall.NoopFirewallDriver
scheduler_max_attempts = 10
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
allow_resize_to_same_host = true
compute_driver = libvirt.LibvirtDriver
my_ip = 10.30.118.91
osapi_compute_workers = 1

[vnc]
novncproxy_host = 10.30.118.91
novncproxy_port = 6080
vncserver_listen = 10.30.118.91
vncserver_proxyclient_address = 10.30.118.91
novncproxy_base_url = http://10.30.118.89:6080/vnc_auto.html

[oslo_messaging_rabbit]
rabbit_userid = openstack
rabbit_password = 5ivsak1tEhOrvCewr6caU8Xg04xWFwIO1hbZe15Y
rabbit_ha_queues = true
rabbit_hosts = 10.30.118.91:5672

[oslo_concurrency]
lock_path = /var/lib/nova/tmp

[glance]
api_servers = 10.30.118.91:9292
num_retries = 1

[cinder]
catalog_info = volume:cinder:internalURL

[neutron]
url = http://10.30.118.89:9696
auth_strategy = keystone
metadata_proxy_shared_secret = FoS1UWFNga9juw9XOEa62DrsgoSj960tduJg6UZ1
service_metadata_proxy = true
auth_url = http://10.30.118.89:35357
auth_type = password
project_domain_name = default
user_domain_id = default
project_name = service
username = neutron
password = kPjcU315lU94x2idvI692HjZV0qFbjI4yhEBfBGE

[database]
connection = mysql+pymysql://nova:mO18VsfPTW3ZOelt1Yn3xDVpy2eGa6z6KaMz7wzp@10.30.118.89/nova
max_pool_size = 50
max_overflow = 1000
max_retries = -1

[api_database]
connection = mysql+pymysql://nova_api:bIBoTWhSmhFiAEeUeY1MdMHMrSzF0vaIZIErgJ0d@10.30.118.89/nova_api
max_retries = -1

[cache]
backend = oslo_cache.memcache_pool
enabled = True
memcache_servers = 10.30.118.91:11211

[keystone_authtoken]
auth_uri = http://10.30.118.89:5000
auth_url = http://10.30.118.89:35357
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = nova
password = vS9aqC21KMVpg80l23qsxDwQY2rLzCvaMAZ5A9t1
memcache_security_strategy = ENCRYPT
memcache_secret_key = CSMv7icC6gAvsCODlj2pBzDPqYPuqMGpnnZaFbhj
memcached_servers = 10.30.118.91:11211

[libvirt]
connection_uri = "qemu+tcp://10.30.118.91/system"

[upgrade_levels]
compute = auto

[oslo_messaging_notifications]
driver = noop

[conductor]
workers = 1

Code:
root@k03-13:~/kolla# git log --oneline
f10a625 Merge "Rabbitmq Upgrade restarts container only when image was changed." into stable/mitaka
74a1487 Add TLS to Kibana Web Interface
...

[Hui Kang (huikang27)]

Hi, Daneyon, are you using the master or stable/mitaka? Thanks. - Hui

[Hui Kang (huikang27)]

I could not boot nova vm with ubuntu source build for the master branch.

The error message I saw at the nova-compute container is http://paste.openstack.org/show/495250/

Looks like some thing related to libvirt and kvm in the nova.conf

[Hui Kang (huikang27)]

By the way, kvm module is installed on the compute node by

modprobe kvm

root@compute-01:~# lsmod | grep kvm
kvm 512000 0

[Hui Kang (huikang27)]

After a second look, I found that the root cause for me is from the nova_libvirt container. I am booting a nova instance in a nested environment, which has no vmx enabled.

After I change virt_type to qemu, nova instance can be created.

[Christian Berendt (berendt)]

Looks like this issue was solved by setting virt_type to qemu. Closing as invalid because this is not a bug.