Ubuntu 4.0.1.0-35~mitaka :: Nova compute error :: VM creation fails.

[Ritam Gangopadhyay]

Setup:-

nodem14 -- 10.204.216.103 -- contrail-controller, contrail-analytics, contrail-analyticsdb, opensatck
nodem6 -- 10.204.216.95 -- contrail-controller, contrail-analytics, contrail-analyticsdb, opensatck
nodem7 -- 10.204.216.96 -- contrail-controller, contrail-analytics, contrail-analyticsdb, opensatck
nodem8 -- 10.204.216.97 -- contrail-compute
nodem9 -- 10.204.216.98 -- contrail-compute
nodec35 -- 10.204.216.160 -- contrail-lb
nodem10 -- 10.204.216.99 -- contrail-compute

nova-compute

2017-08-16 16:26:09.610 4323 WARNING nova.servicegroup.drivers.db [-] Lost connection to nova-conductor for reporting service status.
2017-08-16 16:26:09.611 4323 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 45.00 sec
2017-08-16 16:27:09.617 4323 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 45.01 sec
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task [req-d7955cb2-25c8-43e4-9c38-57f274bf9ad9 - - - - -] Error during ComputeManager.update_available_resource
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task Traceback (most recent call last):
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task task(self, context)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6450, in update_available_resource
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task use_slave=True)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6486, in _get_compute_nodes_in_db
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task use_slave=use_slave)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 174, in wrapper
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task args, kwargs)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task args=args, kwargs=kwargs)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task retry=self.retry)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task timeout=timeout, retry=retry)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task retry=retry)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task result = self._waiter.wait(msg_id, timeout)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task message = self.waiters.get(msg_id, timeout=timeout)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task 'to message ID %s' % msg_id)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task MessagingTimeout: Timed out waiting for a reply to message ID 92068b9955b840f2a9268c2a1c19fae6
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task
2017-08-16 16:28:09.623 4323 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 45.01 sec
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task [req-d7955cb2-25c8-43e4-9c38-57f274bf9ad9 - - - - -] Error during ComputeManager._sync_scheduler_instance_info
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task Traceback (most recent call last):
2017-08-16 16:26:09.610 4323 WARNING nova.servicegroup.drivers.db [-] Lost connection to nova-conductor for reporting service status.
2017-08-16 16:26:09.611 4323 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 45.00 sec
2017-08-16 16:27:09.617 4323 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 45.01 sec
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task [req-d7955cb2-25c8-43e4-9c38-57f274bf9ad9 - - - - -] Error during ComputeManager.update_available_resource
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task Traceback (most recent call last):
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task task(self, context)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6450, in update_available_resource
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task use_slave=True)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6486, in _get_compute_nodes_in_db
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task use_slave=use_slave)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 174, in wrapper
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task args, kwargs)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task args=args, kwargs=kwargs)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task retry=self.retry)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task timeout=timeout, retry=retry)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task retry=retry)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task result = self._waiter.wait(msg_id, timeout)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task message = self.waiters.get(msg_id, timeout=timeout)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task 'to message ID %s' % msg_id)
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task MessagingTimeout: Timed out waiting for a reply to message ID 92068b9955b840f2a9268c2a1c19fae6
2017-08-16 16:27:10.770 4323 ERROR oslo_service.periodic_task
2017-08-16 16:28:09.623 4323 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 45.01 sec
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task [req-d7955cb2-25c8-43e4-9c38-57f274bf9ad9 - - - - -] Error during ComputeManager._sync_scheduler_instance_info
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task Traceback (most recent call last):
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task task(self, context)
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1812, in _sync_scheduler_instance_info
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task use_slave=True)
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 174, in wrapper
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task args, kwargs)
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task args=args, kwargs=kwargs)
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task retry=self.retry)
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task timeout=timeout, retry=retry)
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task retry=retry)
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task result = self._waiter.wait(msg_id, timeout)
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task message = self.waiters.get(msg_id, timeout=timeout)
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task 'to message ID %s' % msg_id)
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task MessagingTimeout: Timed out waiting for a reply to message ID e82d719651ef4266bcfd47caef9fc6ce
2017-08-16 16:28:10.777 4323 ERROR oslo_service.periodic_task

[Rudra Rugge (rrugge)]

I believe Senthil is triaging this currently.

[Senthilnathan Murugappan (msenthil)]

The cluster was reimaged.
If you observe it again, can you take a snapshot of the rabbitmq cluster/queues info and the /var/log/ dump from all rabbitmq nodes and the openstack nodes and the respective compute node.

[Ritam Gangopadhyay (ritam)]

Not able to reproduce.