comparing datetime to None in update-time handling

Bug #1096719 reported by Robert Collins
28
This bug affects 6 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Robert Collins
Folsom
Fix Released
High
Pádraig Brady

Bug Description

2013-01-05 08:02:23 WARNING nova.scheduler.manager [req-c243dee7-fc00-42d8-8654-603b899d471b admin demo] Failed to schedule_run_instance: can't compare datetime.datetime to NoneType
2013-01-05 08:02:23 WARNING nova.scheduler.manager [req-c243dee7-fc00-42d8-8654-603b899d471b admin demo] [instance: 6e4d25bd-9ad8-4e92-b5b4-00fabeed477d] Setting instance to ERROR state.
2013-01-05 08:02:23 ERROR nova.openstack.common.rpc.amqp [req-c243dee7-fc00-42d8-8654-603b899d471b admin demo] Exception during message handling
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 277, in _process_data
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 147, in dispatch
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/manager.py", line 121, in run_instance
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp context, ex, request_spec)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/manager.py", line 109, in run_instance
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp requested_networks, is_first_time, filter_properties)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 60, in schedule_run_instance
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp filter_properties, instance_uuids)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 275, in _schedule
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/host_manager.py", line 386, in get_all_host_states
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp host_state.update_from_compute_node(compute)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/host_manager.py", line 143, in update_from_compute_node
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp if self.updated and self.updated > compute['updated_at']:
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp TypeError: can't compare datetime.datetime to NoneType

this is from HEAD + pending baremetal patches.

Tags: baremetal
Changed in nova:
assignee: nobody → Robert Collins (lifeless)
status: Triaged → Fix Committed
Revision history for this message
Haiwei Xu (xu-haiwei) wrote :

I met the same question when i used the "nova boot" command.

Revision history for this message
Haiwei Xu (xu-haiwei) wrote :
Download full text (4.4 KiB)

I have set tow nodes: precise64 and slave101, and both of them seems to work properly.

vagrant@precise64:~$ nova-manage service list
Binary Host Zone Status State Updated_At
nova-network slave101 nova enabled :-) 2013-01-08 05:28:24
nova-compute slave101 nova enabled :-) 2013-01-08 05:28:24
nova-conductor precise64 nova enabled :-) 2013-01-08 05:28:22
nova-cert precise64 nova enabled :-) 2013-01-08 05:28:24
nova-compute precise64 nova enabled :-) 2013-01-08 05:28:24
nova-network precise64 nova enabled :-) 2013-01-08 05:28:26
nova-scheduler precise64 nova enabled :-) 2013-01-08 05:28:26
nova-consoleauth precise64 nova enabled :-) 2013-01-08 05:28:24

when i try to create an instance with "nova boot",the error happens:
vagrant@precise64:~$ nova boot --hint force_hosts=slave101 --flavor 1 --image a566bbb7-ef41-4fea-8f60-157d86873b22 image01

2013-01-08 05:29:17 DEBUG nova.scheduler.filter_scheduler [req-dff32077-fe8c-49d9-8784-25be8f692d8a admin demo] Attempting to build 1 instance(s) from (pid=9080) schedule_run_instance /opt/stack/nova/nova/scheduler/filter_scheduler.py:53
2013-01-08 05:29:17 WARNING nova.scheduler.manager [req-dff32077-fe8c-49d9-8784-25be8f692d8a admin demo] Failed to schedule_run_instance: can't compare datetime.datetime to NoneType
2013-01-08 05:29:17 WARNING nova.scheduler.manager [req-dff32077-fe8c-49d9-8784-25be8f692d8a admin demo] [instance: 631f69ff-9827-427f-a5d0-e59359687519] Setting instance to ERROR state.
2013-01-08 05:29:17 ERROR nova.openstack.common.rpc.amqp [req-dff32077-fe8c-49d9-8784-25be8f692d8a admin demo] Exception during message handling
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 277, in _process_data
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 147, in dispatch
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/manager.py", line 121, in run_instance
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp context, ex, request_spec)
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/manager.py", line 109, in run_...

Read more...

Revision history for this message
Haiwei Xu (xu-haiwei) wrote :

hi,

i have solved this problem by editing the /etc/nova/nova.conf

I deleted the line "allow_admin_api=True"

Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-2
status: Fix Committed → Fix Released
Revision history for this message
Joshua (josh-ni1sr89) wrote :

I have the exact same error, I did not have "allow_admin_api=True" in my nova.conf. Can someone point me to the code changes for this fix?

Revision history for this message
Richard Knight (richmgb) wrote :
Revision history for this message
Ryan Richard (ryan-richard) wrote :

This bug also appears to affect Folsom with the .1 backport branch.

I also confirmed that the patch appears to fix the issue.

Revision history for this message
Lorin Hochstein (lorinh) wrote :

I hit this in Folsom as well, any chance of a backport to fix this?

Revision history for this message
Robert Collins (lifeless) wrote :

I seem to recall someone backporting this in the last week or so.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/folsom)

Reviewed: https://review.openstack.org/23758
Committed: http://github.com/openstack/nova/commit/5b43cef510b68cff1f6e2f80742d3204b0b51e45
Submitter: Jenkins
Branch: stable/folsom

commit 5b43cef510b68cff1f6e2f80742d3204b0b51e45
Author: Robert Collins <email address hidden>
Date: Mon Jan 7 15:12:39 2013 +1300

    Handle compute node records with no timestamp.

    Commit 94560ab57d9fc23673f42017e6f2a78cb2b66b7a added a fast-path
    to avoid work when baremetal compute nodes haven't changed, but if
    the compute message has None for its updated-at timestamp (which
    can happen during startup) the comparison crashes.

    Fixes bug: 1096719
    Change-Id: Ib8add431d975eee133deefea62eb53479a97396e
    (cherry picked from commit fad69df25ffcea2a44cbf3ef636a68863a2d64d9)

Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-2 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers