Contrail Networking :: ubuntu 16.04 build 29 ocata :: nova conductor fails while creating VM.

[Ritam Gangopadhyay]

In a contrail networking setup, where opensatck is brought up using contrail-cloud image and contrail-services are brought up using networking image. I see nova conductor fails while trying to find host to schedule building a VM and reports error saying:-

"NoValidHost: No valid host was found. There are not enough hosts available."

While availability-zones and cova compute services are listed perfectly from the openstack and nova cli.

Setup:- R4.0.1.0 build 29 Ocata multi node setup.

nodei19 10.204.217.131 openstack
nodec28 10.204.217.13 controller, analytics, analyticsdb
nodec10 10.204.217.176 controller, analytics, analyticsdb
nodec33 10.204.217.168 controller, analytics, analyticsdb
nodeg37 10.204.217.77 lb
nodei17 10.204.217.129 compute
nodei20 10.204.217.132 compute

***************************************
***************************************

nova conductor log

2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager [req-f2d02463-9b5f-4f9a-ba08-0b8bea0275ee 9860fb722d284aa5a350d287fd4078d0 65fc7a0856114310a84c8b67025a9aee - - -] Failed to schedule instances
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 866, in schedule_and_build_instances
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 597, in _schedule_instances
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager retry=self.retry)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 458, in send
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager retry=retry)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 449, in _send
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager raise result
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 218, in inner
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 98, in select_destinations
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager dests = self.driver.select_destinations(ctxt, spec_obj)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager raise exception.NoValidHost(reason=reason)
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager NoValidHost: No valid host was found. There are not enough hosts available.
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager
2017-09-10 01:28:20.459 19 ERROR nova.conductor.manager
2017-09-10 01:28:20.588 19 WARNING nova.scheduler.utils [req-f2d02463-9b5f-4f9a-ba08-0b8bea0275ee 9860fb722d284aa5a350d287fd4078d0 65fc7a0856114310a84c8b67025a9aee - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 218, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 98, in select_destinations
    dests = self.driver.select_destinations(ctxt, spec_obj)

  File "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line 79, in select_destinations
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.

2017-09-10 01:28:20.588 19 WARNING nova.scheduler.utils [req-f2d02463-9b5f-4f9a-ba08-0b8bea0275ee 9860fb722d284aa5a350d287fd4078d0 65fc7a0856114310a84c8b67025a9aee - - -] [instance: 4ebea18a-1009-46fd-b256-285d267e340b] Setting instance to ERROR state.

*******************************************************************
*******************************************************************
nova-compute service and availability zones
*******************************************************************
*******************************************************************

root@nodei19:~# nova service-list
/usr/lib/python2.7/dist-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
  warnings.warn(msg)
+----+------------------+---------+----------+----------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+---------+----------+----------+-------+----------------------------+-----------------+
| 1 | nova-scheduler | nodei19 | internal | enabled | up | 2017-09-10T05:22:55.000000 | - |
| 5 | nova-conductor | nodei19 | internal | enabled | up | 2017-09-10T05:22:55.000000 | - |
| 8 | nova-consoleauth | nodei19 | internal | enabled | up | 2017-09-10T05:23:01.000000 | - |
| 9 | nova-compute | nodei19 | nova | disabled | up | 2017-09-10T05:22:54.000000 | - |
| 10 | nova-compute | nodei17 | nova | enabled | up | 2017-09-10T05:22:51.000000 | - |
| 11 | nova-compute | nodei20 | nova | enabled | up | 2017-09-10T05:22:50.000000 | - |
+----+------------------+---------+----------+----------+-------+----------------------------+-----------------+
root@nodei19:~# nova availability-zone-list
/usr/lib/python2.7/dist-packages/novaclient/client.py:278: UserWarning: The 'tenant_id' argument is deprecated in Ocata and its use may result in errors in future releases. As 'project_id' is provided, the 'tenant_id' argument will be ignored.
  warnings.warn(msg)
+-----------------------+----------------------------------------+
| Name | Status |
+-----------------------+----------------------------------------+
| internal | available |
| |- nodei19 | |
| | |- nova-conductor | enabled :-) 2017-09-10T05:23:15.000000 |
| | |- nova-scheduler | enabled :-) 2017-09-10T05:23:15.000000 |
| | |- nova-consoleauth | enabled :-) 2017-09-10T05:23:11.000000 |
| nova | available |
| |- nodei17 | |
| | |- nova-compute | enabled :-) 2017-09-10T05:23:06.000000 |
| |- nodei20 | |
| | |- nova-compute | enabled :-) 2017-09-10T05:23:05.000000 |
+-----------------------+----------------------------------------+
root@nodei19:~# openstack availability zone list
+-----------+-------------+
| Zone Name | Zone Status |
+-----------+-------------+
| internal | available |
| nova | available |
+-----------+-------------+
root@nodei19:~#

[Ritam Gangopadhyay (ritam)]

The Same error is seen with CB build 40. verified while creating VM on CB build 40.

[Nitish Krishna Kaveri (nitishk)]

I tried to provision contrail_cluster with networking image but it was missing a package -

This package seems to be missing from networking tgz:

root@nk-vm6:~# apt-cache policy python-osprofiler
python-osprofiler:
  Installed: (none)
  Candidate: (none)
  Version table:

aptitude install of nova-compute complains:

The following packages have unmet dependencies:
 python-nova : Depends: python-osprofiler which is a virtual package and is not provided by any available package.

 libhogweed4 : Depends: libnettle6 (= 3.2-1ubuntu0.16.04.1) but 3.3-1~cloud0 is to be installed.

When I provisioned both using the cloud image I was able to spawn the VMS:

root@nk-vm1:~# openstack server list
+--------------------------------------+------+--------+---------------------+------------+
| ID | Name | Status | Networks | Image Name |
+--------------------------------------+------+--------+---------------------+------------+
| 8f548310-387f-4c3d-86ab-161b61d721c5 | vm2 | ACTIVE | demo-net=30.30.30.4 | cirros |
| 48f8be82-1519-4001-a2d2-2423bb8ad11f | vm1 | ACTIVE | demo-net=30.30.30.3 | cirros |
+--------------------------------------+------+--------+---------------------+------------+

[Nitish Krishna Kaveri (nitishk)]

Marking as incomplete.
Will take a look if reproduced in Sanity setup.

Nagendra will take a look at the issue with "python-osprofiler"

[Ritam Gangopadhyay (ritam)]

Re-created and re-opening the bug on build 30, setup mentioned in the bug is in failed state. Below are fresh logs from the failure on build 30. Also horizon screenshot of error while creating VM is attached.

python-osprofiler is not an issue at all and is seen because of wrong packages used to provision. This is not seen with ocata networking image on build 30 or earlier CB build 40.

Nova Conductor Logs:-
************************
************************
************************

2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager [req-65d8be3b-52ca-4007-9d7f-e0e2f94f3016 6ff378dec4314b1bbbfdc9994302ee15 ded86217372445adba1de3585b5ac14c - - -] Failed to schedule instances
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager Traceback (most recent call last):
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 866, in schedule_and_build_instances
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager request_specs[0].to_legacy_filter_properties_dict())
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 597, in _schedule_instances
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager hosts = self.scheduler_client.select_destinations(context, spec_obj)
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", line 371, in wrapped
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager return func(*args, **kwargs)
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager return self.queryclient.select_destinations(context, spec_obj)
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 32, in select_destinations
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager return self.scheduler_rpcapi.select_destinations(context, spec_obj)
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 129, in select_destinations
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager retry=self.retry)
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
2017-09-13 12:43:08.582 19 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2017-09-13 12:43:08.582 19 ERRO...

[Nitish Krishna Kaveri (nitishk)]

From Ram:

Hi Ritam,

I took a quick look and noticed that the neutron-metadata agent has the shared secret set :

root@nodei19:~# cat /etc/kolla/neutron-metadata-agent/metadata_agent.ini
[DEFAULT]
nova_metadata_ip = 192.168.100.20
nova_metadata_port = 8775
metadata_proxy_shared_secret = 95B92A96BBBE

whereas in nova.conf, we have not set it:

root@nodei17:~# cat /etc/nova/nova.conf | grep metadata_proxy_shared_secret
# * metadata_proxy_shared_secret
#metadata_proxy_shared_secret =

Please add this section under “openstack” section in contrail_cluster JSON:

                        "neutron": {
                                 "shared_secret": "95B92A96BBBE "
                        },

Also transport_url is also not set in nova.conf on the compute. This wil be populated when openstack_sku is set to ‘ocata’ in the inventory. I do not see this variable in the inventory. Can you try setting this variable in the contrail_cluster JSON under “contrail_4” section and reprovision the contrail_cluster?

                       “contrail_4”: {
                           ….
                           “openstack_sku”: “ocata”
                       }

Ritam is confirming this fix

[Ritam Gangopadhyay (ritam)]

adding the new configuration of shared_secret and metadata secret under contrail and openstack cluster
adding configuration openstack sku under contrail_4 under contrail cluster

with above changes, I was able to provision successfully and nova conductor error is no longer seen while spawning VM's successfully.