############## nova-compute log for node with the vms 2012-02-27 17:35:26,707 INFO nova.rpc [43e0bcc2-2048-4875-b289-0270d1973c0a tester testproject] Connected to AMQP server on 172.18.0.131:5672 2012-02-27 17:35:27,177 DEBUG nova.rpc [6b0097dc-e3c9-482d-8002-69d57f2b7d8c tester testproject] Making asynchronous cast on network... from (pid=1206) cast /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:756 2012-02-27 17:35:27,179 DEBUG nova.rpc [611af1b6-bdcc-4779-83a3-1f7fcbdbad22 tester testproject] Making asynchronous cast on network... from (pid=1206) cast /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:756 2012-02-27 17:35:27,228 ERROR nova.rpc [6b0097dc-e3c9-482d-8002-69d57f2b7d8c tester testproject] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 100, in wrapped (nova.rpc): TRACE: return f(*args, **kw) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 118, in decorated_function (nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 524, in terminate_instance (nova.rpc): TRACE: self._shutdown_instance(context, instance_id, 'Terminating') (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 514, in _shutdown_instance (nova.rpc): TRACE: ' instance: %s') % instance_id) (nova.rpc): TRACE: Error: trying to destroy already destroyed instance: 155 (nova.rpc): TRACE: 2012-02-27 17:35:27,273 ERROR nova.rpc [611af1b6-bdcc-4779-83a3-1f7fcbdbad22 tester testproject] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 100, in wrapped (nova.rpc): TRACE: return f(*args, **kw) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 118, in decorated_function (nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 524, in terminate_instance (nova.rpc): TRACE: self._shutdown_instance(context, instance_id, 'Terminating') (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 514, in _shutdown_instance (nova.rpc): TRACE: ' instance: %s') % instance_id) (nova.rpc): TRACE: Error: trying to destroy already destroyed instance: 155 (nova.rpc): TRACE: 2012-02-27 17:35:27,274 DEBUG nova.rpc [43e0bcc2-2048-4875-b289-0270d1973c0a tester testproject] Making asynchronous cast on network... from (pid=1206) cast /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:756 2012-02-27 17:35:27,275 DEBUG nova.rpc [cd7ba21e-1213-4322-bd8f-2b1fadb48827 tester testproject] Making asynchronous cast on network... from (pid=1206) cast /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:756 2012-02-27 17:35:27,275 DEBUG nova.rpc [b7ca50db-bf6f-44af-adfc-23d2061cf374 tester testproject] Making asynchronous cast on network... from (pid=1206) cast /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:756 2012-02-27 17:35:27,275 DEBUG nova.rpc [f38b1977-a2f5-4b1f-83df-e937977df1bc MedCloud test] Making asynchronous cast on network... from (pid=1206) cast /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:756 2012-02-27 17:35:27,307 ERROR nova.rpc [43e0bcc2-2048-4875-b289-0270d1973c0a tester testproject] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 100, in wrapped (nova.rpc): TRACE: return f(*args, **kw) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 118, in decorated_function (nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 524, in terminate_instance (nova.rpc): TRACE: self._shutdown_instance(context, instance_id, 'Terminating') (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 514, in _shutdown_instance (nova.rpc): TRACE: ' instance: %s') % instance_id) (nova.rpc): TRACE: Error: trying to destroy already destroyed instance: 200 (nova.rpc): TRACE: 2012-02-27 17:35:27,333 ERROR nova.rpc [cd7ba21e-1213-4322-bd8f-2b1fadb48827 tester testproject] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 100, in wrapped (nova.rpc): TRACE: return f(*args, **kw) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 118, in decorated_function (nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 524, in terminate_instance (nova.rpc): TRACE: self._shutdown_instance(context, instance_id, 'Terminating') (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 514, in _shutdown_instance (nova.rpc): TRACE: ' instance: %s') % instance_id) (nova.rpc): TRACE: Error: trying to destroy already destroyed instance: 155 (nova.rpc): TRACE: 2012-02-27 17:35:27,365 ERROR nova.rpc [b7ca50db-bf6f-44af-adfc-23d2061cf374 tester testproject] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 100, in wrapped (nova.rpc): TRACE: return f(*args, **kw) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 118, in decorated_function (nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 524, in terminate_instance (nova.rpc): TRACE: self._shutdown_instance(context, instance_id, 'Terminating') (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 514, in _shutdown_instance (nova.rpc): TRACE: ' instance: %s') % instance_id) (nova.rpc): TRACE: Error: trying to destroy already destroyed instance: 155 (nova.rpc): TRACE: 2012-02-27 17:35:27,416 ERROR nova.rpc [f38b1977-a2f5-4b1f-83df-e937977df1bc MedCloud test] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 100, in wrapped (nova.rpc): TRACE: return f(*args, **kw) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 118, in decorated_function (nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 524, in terminate_instance (nova.rpc): TRACE: self._shutdown_instance(context, instance_id, 'Terminating') (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 514, in _shutdown_instance (nova.rpc): TRACE: ' instance: %s') % instance_id) (nova.rpc): TRACE: Error: trying to destroy already destroyed instance: 200 (nova.rpc): TRACE: 2012-02-27 17:35:27,542 DEBUG nova.rpc [14becee1-46ab-4b09-821d-a9083cd3217d MedCloud test] Making asynchronous cast on network... from (pid=1206) cast /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:756 2012-02-27 17:35:27,543 DEBUG nova.rpc [79fd8125-7a93-463e-b959-34ebdefd2445 tester testproject] Making asynchronous cast on network... from (pid=1206) cast /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:756 2012-02-27 17:35:27,575 ERROR nova.rpc [14becee1-46ab-4b09-821d-a9083cd3217d MedCloud test] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 100, in wrapped (nova.rpc): TRACE: return f(*args, **kw) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 118, in decorated_function (nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 524, in terminate_instance (nova.rpc): TRACE: self._shutdown_instance(context, instance_id, 'Terminating') (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 514, in _shutdown_instance (nova.rpc): TRACE: ' instance: %s') % instance_id) (nova.rpc): TRACE: Error: trying to destroy already destroyed instance: 155 (nova.rpc): TRACE: 2012-02-27 17:35:27,616 ERROR nova.rpc [79fd8125-7a93-463e-b959-34ebdefd2445 tester testproject] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 100, in wrapped (nova.rpc): TRACE: return f(*args, **kw) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 118, in decorated_function (nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 524, in terminate_instance (nova.rpc): TRACE: self._shutdown_instance(context, instance_id, 'Terminating') (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 514, in _shutdown_instance (nova.rpc): TRACE: ' instance: %s') % instance_id) (nova.rpc): TRACE: Error: trying to destroy already destroyed instance: 155 (nova.rpc): TRACE: 2012-02-27 17:35:27,617 DEBUG nova.rpc [5774be29-df01-4129-b9b9-2d7bd05507b5 tester testproject] Making asynchronous cast on network... from (pid=1206) cast /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:756 2012-02-27 17:35:29,078 INFO nova.virt.libvirt.firewall [284a1f06-578f-4c1f-a4cc-76fe18498630 MedCloud test] Attempted to unfilter instance 155 which is not filtered 2012-02-27 17:35:29,078 INFO nova [284a1f06-578f-4c1f-a4cc-76fe18498630 MedCloud test] called setup_basic_filtering in nwfilter 2012-02-27 17:35:29,078 INFO nova [284a1f06-578f-4c1f-a4cc-76fe18498630 MedCloud test] ensuring static filters 2012-02-27 17:35:31,199 INFO nova.virt.libvirt.firewall [05f5fe84-d056-412a-bb82-7ab2b91ff7dd MedCloud test] Attempted to unfilter instance 200 which is not filtered 2012-02-27 17:35:31,199 INFO nova [05f5fe84-d056-412a-bb82-7ab2b91ff7dd MedCloud test] called setup_basic_filtering in nwfilter 2012-02-27 17:35:31,199 INFO nova [05f5fe84-d056-412a-bb82-7ab2b91ff7dd MedCloud test] ensuring static filters 2012-02-27 17:35:32,736 INFO nova.virt.libvirt_conn [-] Instance instance-0000009b destroyed successfully. 2012-02-27 17:35:33,424 INFO nova.virt.libvirt_conn [-] Instance instance-000000c8 destroyed successfully. 2012-02-27 17:35:33,425 DEBUG nova.rpc [121aa6aa-646a-48db-902f-e97c8d33ca7a Med ############## nova-compute log for other node Nothing interesting. ############## nova-network log for node with the vms 2012-02-27 17:35:27,211 DEBUG nova.rpc [-] Pool creating new connection from (pid=1201) create /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:504 2012-02-27 17:35:27,212 DEBUG nova.rpc [5774be29-df01-4129-b9b9-2d7bd05507b5 tester testproject] received {u'_context_roles': [], u'_context_request_id': u'6b0097dc-e3c9-482d-8002-69d57f2b7d8c', u'_context_read_deleted': False, u'args': {u'instance_id': 155, u'project_id': u'test'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'testproject', u'_context_timestamp': u'2012-02-27T22:24:02.402300', u'_context_user_id': u'tester', u'method': u'deallocate_for_instance', u'_context_remote_address': u'127.0.0.1'} from (pid=1201) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:27,213 DEBUG nova.rpc [5774be29-df01-4129-b9b9-2d7bd05507b5 tester testproject] unpacked context: {'user_id': u'tester', 'roles': [], 'timestamp': u'2012-02-27T22:24:02.402300', 'auth_token': None, 'msg_id': None, 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'6b0097dc-e3c9-482d-8002-69d57f2b7d8c', 'project_id': u'testproject', 'read_deleted': False} from (pid=1201) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:27,213 DEBUG nova.network.manager [6b0097dc-e3c9-482d-8002-69d57f2b7d8c tester testproject] floating IP deallocation for instance |155| from (pid=1201) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251 2012-02-27 17:35:27,218 DEBUG nova.network.manager [6b0097dc-e3c9-482d-8002-69d57f2b7d8c tester testproject] network deallocation for instance |155| from (pid=1201) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:465 2012-02-27 17:35:27,294 ERROR nova.rpc [928764eb-1345-4721-8198-19563d9c82f9 None None] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 275, in deallocate_for_instance (nova.rpc): TRACE: super(FloatingIP, self).deallocate_for_instance(context, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 468, in deallocate_for_instance (nova.rpc): TRACE: self.deallocate_fixed_ip(context, fixed_ip['address'], **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 641, in deallocate_fixed_ip (nova.rpc): TRACE: instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 404, in _do_trigger_security_group_members_refresh_for_instance (nova.rpc): TRACE: instance_ref = self.db.instance_get(admin_context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 500, in instance_get (nova.rpc): TRACE: return IMPL.instance_get(context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 119, in wrapper (nova.rpc): TRACE: return f(*args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1171, in instance_get (nova.rpc): TRACE: raise exception.InstanceNotFound(instance_id=instance_id) (nova.rpc): TRACE: InstanceNotFound: Instance 155 could not be found. (nova.rpc): TRACE: 2012-02-27 17:35:27,308 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'43e0bcc2-2048-4875-b289-0270d1973c0a', u'_context_read_deleted': False, u'args': {u'instance_id': 200, u'project_id': u'test'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'testproject', u'_context_timestamp': u'2012-02-27T22:30:42.989849', u'_context_user_id': u'tester', u'method': u'deallocate_for_instance', u'_context_remote_address': u'127.0.0.1'} from (pid=1201) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:27,308 DEBUG nova.rpc [-] unpacked context: {'user_id': u'tester', 'roles': [], 'timestamp': u'2012-02-27T22:30:42.989849', 'auth_token': None, 'msg_id': None, 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'43e0bcc2-2048-4875-b289-0270d1973c0a', 'project_id': u'testproject', 'read_deleted': False} from (pid=1201) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:27,309 DEBUG nova.rpc [43e0bcc2-2048-4875-b289-0270d1973c0a tester testproject] received {u'_context_roles': [], u'_context_request_id': u'b7ca50db-bf6f-44af-adfc-23d2061cf374', u'_context_read_deleted': False, u'args': {u'instance_id': 155, u'project_id': u'test'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'testproject', u'_context_timestamp': u'2012-02-27T22:25:19.338792', u'_context_user_id': u'tester', u'method': u'deallocate_for_instance', u'_context_remote_address': u'127.0.0.1'} from (pid=1201) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:27,309 DEBUG nova.rpc [43e0bcc2-2048-4875-b289-0270d1973c0a tester testproject] unpacked context: {'user_id': u'tester', 'roles': [], 'timestamp': u'2012-02-27T22:25:19.338792', 'auth_token': None, 'msg_id': None, 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'b7ca50db-bf6f-44af-adfc-23d2061cf374', 'project_id': u'testproject', 'read_deleted': False} from (pid=1201) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:27,309 DEBUG nova.network.manager [43e0bcc2-2048-4875-b289-0270d1973c0a tester testproject] floating IP deallocation for instance |200| from (pid=1201) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251 2012-02-27 17:35:27,314 DEBUG nova.network.manager [43e0bcc2-2048-4875-b289-0270d1973c0a tester testproject] network deallocation for instance |200| from (pid=1201) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:465 2012-02-27 17:35:27,445 ERROR nova.rpc [4a59acea-7ec9-4eff-b346-7afcae98cfbc None None] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 275, in deallocate_for_instance (nova.rpc): TRACE: super(FloatingIP, self).deallocate_for_instance(context, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 468, in deallocate_for_instance (nova.rpc): TRACE: self.deallocate_fixed_ip(context, fixed_ip['address'], **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 641, in deallocate_fixed_ip (nova.rpc): TRACE: instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 404, in _do_trigger_security_group_members_refresh_for_instance (nova.rpc): TRACE: instance_ref = self.db.instance_get(admin_context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 500, in instance_get (nova.rpc): TRACE: return IMPL.instance_get(context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 119, in wrapper (nova.rpc): TRACE: return f(*args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1171, in instance_get (nova.rpc): TRACE: raise exception.InstanceNotFound(instance_id=instance_id) (nova.rpc): TRACE: InstanceNotFound: Instance 200 could not be found. (nova.rpc): TRACE: 2012-02-27 17:35:27,446 DEBUG nova.network.manager [b7ca50db-bf6f-44af-adfc-23d2061cf374 tester testproject] floating IP deallocation for instance |155| from (pid=1201) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251 2012-02-27 17:35:27,451 DEBUG nova.network.manager [b7ca50db-bf6f-44af-adfc-23d2061cf374 tester testproject] network deallocation for instance |155| from (pid=1201) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:465 2012-02-27 17:35:27,528 ERROR nova.rpc [71f2ba08-0d93-457f-8c69-9860400ee7f5 None None] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 275, in deallocate_for_instance (nova.rpc): TRACE: super(FloatingIP, self).deallocate_for_instance(context, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 468, in deallocate_for_instance (nova.rpc): TRACE: self.deallocate_fixed_ip(context, fixed_ip['address'], **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 641, in deallocate_fixed_ip (nova.rpc): TRACE: instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 404, in _do_trigger_security_group_members_refresh_for_instance (nova.rpc): TRACE: instance_ref = self.db.instance_get(admin_context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 500, in instance_get (nova.rpc): TRACE: return IMPL.instance_get(context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 119, in wrapper (nova.rpc): TRACE: return f(*args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1171, in instance_get (nova.rpc): TRACE: raise exception.InstanceNotFound(instance_id=instance_id) (nova.rpc): TRACE: InstanceNotFound: Instance 155 could not be found. (nova.rpc): TRACE: 2012-02-27 17:35:27,530 INFO nova.rpc [-] Connected to AMQP server on 172.18.0.131:5672 2012-02-27 17:35:27,530 INFO nova.rpc [-] Connected to AMQP server on 172.18.0.131:5672 2012-02-27 17:35:27,531 INFO nova.rpc [-] Connected to AMQP server on 172.18.0.131:5672 2012-02-27 17:35:27,531 INFO nova.rpc [-] Connected to AMQP server on 172.18.0.131:5672 2012-02-27 17:35:27,531 INFO nova.rpc [-] Connected to AMQP server on 172.18.0.131:5672 2012-02-27 17:35:27,532 INFO nova.rpc [-] Connected to AMQP server on 172.18.0.131:5672 2012-02-27 17:35:27,543 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'14becee1-46ab-4b09-821d-a9083cd3217d', u'_context_read_deleted': False, u'args': {u'instance_id': 155, u'project_id': u'test'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'test', u'_context_timestamp': u'2012-02-27T22:21:47.516372', u'_context_user_id': u'MedCloud', u'method': u'deallocate_for_instance', u'_context_remote_address': u'127.0.0.1'} from (pid=1201) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:27,544 DEBUG nova.rpc [-] unpacked context: {'user_id': u'MedCloud', 'roles': [], 'timestamp': u'2012-02-27T22:21:47.516372', 'auth_token': None, 'msg_id': None, 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'14becee1-46ab-4b09-821d-a9083cd3217d', 'project_id': u'test', 'read_deleted': False} from (pid=1201) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:27,544 DEBUG nova.network.manager [14becee1-46ab-4b09-821d-a9083cd3217d MedCloud test] floating IP deallocation for instance |155| from (pid=1201) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251 2012-02-27 17:35:27,549 DEBUG nova.network.manager [14becee1-46ab-4b09-821d-a9083cd3217d MedCloud test] network deallocation for instance |155| from (pid=1201) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:465 2012-02-27 17:35:27,634 ERROR nova.rpc [5347d8a3-e2e4-4658-9547-c757474e39fb None None] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 275, in deallocate_for_instance (nova.rpc): TRACE: super(FloatingIP, self).deallocate_for_instance(context, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 468, in deallocate_for_instance (nova.rpc): TRACE: self.deallocate_fixed_ip(context, fixed_ip['address'], **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 641, in deallocate_fixed_ip (nova.rpc): TRACE: instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 404, in _do_trigger_security_group_members_refresh_for_instance (nova.rpc): TRACE: instance_ref = self.db.instance_get(admin_context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 500, in instance_get (nova.rpc): TRACE: return IMPL.instance_get(context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 119, in wrapper (nova.rpc): TRACE: return f(*args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1171, in instance_get (nova.rpc): TRACE: raise exception.InstanceNotFound(instance_id=instance_id) (nova.rpc): TRACE: InstanceNotFound: Instance 155 could not be found. (nova.rpc): TRACE: 2012-02-27 17:35:34,737 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'5774be29-df01-4129-b9b9-2d7bd05507b5', u'_context_read_deleted': False, u'args': {u'instance_id': 155, u'project_id': u'test'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'testproject', u'_context_timestamp': u'2012-02-27T22:25:55.566005', u'_context_user_id': u'tester', u'method': u'deallocate_for_instance', u'_context_remote_address': u'127.0.0.1'} from (pid=1201) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:34,737 DEBUG nova.rpc [-] unpacked context: {'user_id': u'tester', 'roles': [], 'timestamp': u'2012-02-27T22:25:55.566005', 'auth_token': None, 'msg_id': None, 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'5774be29-df01-4129-b9b9-2d7bd05507b5', 'project_id': u'testproject', 'read_deleted': False} from (pid=1201) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:34,738 DEBUG nova.network.manager [5774be29-df01-4129-b9b9-2d7bd05507b5 tester testproject] floating IP deallocation for instance |155| from (pid=1201) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251 2012-02-27 17:35:34,743 DEBUG nova.network.manager [5774be29-df01-4129-b9b9-2d7bd05507b5 tester testproject] network deallocation for instance |155| from (pid=1201) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:465 2012-02-27 17:35:34,831 ERROR nova.rpc [9ac29291-3997-4612-acdf-c0676890e80a None None] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 275, in deallocate_for_instance (nova.rpc): TRACE: super(FloatingIP, self).deallocate_for_instance(context, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 468, in deallocate_for_instance (nova.rpc): TRACE: self.deallocate_fixed_ip(context, fixed_ip['address'], **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 641, in deallocate_fixed_ip (nova.rpc): TRACE: instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 404, in _do_trigger_security_group_members_refresh_for_instance (nova.rpc): TRACE: instance_ref = self.db.instance_get(admin_context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 500, in instance_get (nova.rpc): TRACE: return IMPL.instance_get(context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 119, in wrapper (nova.rpc): TRACE: return f(*args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1171, in instance_get (nova.rpc): TRACE: raise exception.InstanceNotFound(instance_id=instance_id) (nova.rpc): TRACE: InstanceNotFound: Instance 155 could not be found. (nova.rpc): TRACE: 2012-02-27 17:35:54,085 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'c23b8d1b-201d-4b6a-ae70-87bdbdac90c4', u'_context_read_deleted': False, u'args': {u'address': u'10.0.0.25'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-02-27T22:35:54.046664', u'_context_user_id': None, u'method': u'lease_fixed_ip', u'_context_remote_address': None} from (pid=1201) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:54,085 DEBUG nova.rpc [-] unpacked context: {'user_id': None, 'roles': [], 'timestamp': u'2012-02-27T22:35:54.046664', 'auth_token': None, 'msg_id': None, 'remote_address': None, 'strategy': u'noauth', 'is_admin': True, 'request_id': u'c23b8d1b-201d-4b6a-ae70-87bdbdac90c4', 'project_id': None, 'read_deleted': False} from (pid=1201) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:54,086 DEBUG nova.network.manager [c23b8d1b-201d-4b6a-ae70-87bdbdac90c4 None None] Leased IP |10.0.0.25| from (pid=1201) lease_fixed_ip /usr/lib/python2.7/dist-packages/nova/network/manager.py:650 2012-02-27 17:35:54,202 WARNING nova.network.manager [c23b8d1b-201d-4b6a-ae70-87bdbdac90c4 None None] IP |10.0.0.25| leased that isn't allocated 2012-02-27 17:35:58,059 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'29324242-e780-4d84-a4af-3a6991139b12', u'_context_read_deleted': False, u'args': {u'address': u'10.0.0.39'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-02-27T22:35:58.022322', u'_context_user_id': None, u'method': u'lease_fixed_ip', u'_context_remote_address': None} from (pid=1201) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:58,060 DEBUG nova.rpc [-] unpacked context: {'user_id': None, 'roles': [], 'timestamp': u'2012-02-27T22:35:58.022322', 'auth_token': None, 'msg_id': None, 'remote_address': None, 'strategy': u'noauth', 'is_admin': True, 'request_id': u'29324242-e780-4d84-a4af-3a6991139b12', 'project_id': None, 'read_deleted': False} from (pid=1201) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:58,060 DEBUG nova.network.manager [29324242-e780-4d84-a4af-3a6991139b12 None None] Leased IP |10.0.0.39| from (pid=1201) lease_fixed_ip /usr/lib/python2.7/dist-packages/nova/network/manager.py:650 2012-02-27 17:35:58,144 WARNING nova.network.manager [29324242-e780-4d84-a4af-3a6991139b12 None None] IP |10.0.0.39| leased that isn't allocated 2012-02-27 17:36:40,093 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'5129e90f-175f-4b8b-8716-a500645bcd86', u'_context_read_deleted': False, u'args': {u'address': u'10.0.0.25'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-02-27T22:36:40.055409', u'_context_user_id': None, u'method': u'lease_fixed_ip', u'_context_remote_address': None} from (pid=1201) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:36:40,094 DEBUG nova.rpc [-] unpacked context: {'user_id': None, 'roles': [], 'timestamp': u'2012-02-27T22:36:40.055409', 'auth_token': None, 'msg_id': None, 'remote_address': None, 'strategy': u'noauth', 'is_admin': True, 'request_id': u'5129e90f-175f-4b8b-8716-a500645bcd86', 'project_id': None, 'read_deleted': False} from (pid=1201) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:36:40,094 DEBUG nova.network.manager [5129e90f-175f-4b8b-8716-a500645bcd86 None None] Leased IP |10.0.0.25| from (pid=1201) lease_fixed_ip /usr/lib/python2.7/dist-packages/nova/network/manager.py:650 2012-02-27 17:36:40,177 WARNING nova.network.manager [5129e90f-175f-4b8b-8716-a500645bcd86 None None] IP |10.0.0.25| leased that isn't allocated 2012-02-27 17:36:51,883 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'91185bb5-ea71-49ef-aad1-b2b7c5e3c717', u'_context_read_deleted': False, u'args': {u'address': u'10.0.0.39'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-02-27T22:36:51.846067', u'_context_user_id': None, u'method': u'lease_fixed_ip', u'_context_remote_address': None} from (pid=1201) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:36:51,883 DEBUG nova.rpc [-] unpacked context: {'user_id': None, 'roles': [], 'timestamp': u'2012-02-27T22:36:51.846067', 'auth_token': None, 'msg_id': None, 'remote_address': None, 'strategy': u'noauth', 'is_admin': True, 'request_id': u'91185bb5-ea71-49ef-aad1-b2b7c5e3c717', 'project_id': None, 'read_deleted': False} from (pid=1201) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:36:51,884 DEBUG nova.network.manager [91185bb5-ea71-49ef-aad1-b2b7c5e3c717 None None] Leased IP |10.0.0.39| from (pid=1201) lease_fixed_ip /usr/lib/python2.7/dist-packages/nova/network/manager.py:650 2012-02-27 17:36:51,979 WARNING nova.network.manager [91185bb5-ea71-49ef-aad1-b2b7c5e3c717 None None] IP |10.0.0.39| leased that isn't allocated 2012-02-27 17:37:33,407 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'be0218cd-335b-45d9-82dc-71cf06d1ac93', u'_context_read_deleted': False, u'args': {u'address': u'10.0.0.25'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-02-27T22:37:33.369555', u'_context_user_id': None, u'method': u'lease_fixed_ip', u'_context_remote_address': None} from (pid=1201) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:37:33,408 DEBUG nova.rpc [-] unpacked context: {'user_id': None, 'roles': [], 'timestamp': u'2012-02-27T22:37:33.369555', 'auth_token': None, 'msg_id': None, 'remote_address': None, 'strategy': u'noauth', 'is_admin': True, 'request_id': u'be0218cd-335b-45d9-82dc-71cf06d1ac93', 'project_id': None, 'read_deleted': False} from (pid=1201) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:37:33,408 DEBUG nova.network.manager [be0218cd-335b-45d9-82dc-71cf06d1ac93 None None] Leased IP |10.0.0.25| from (pid=1201) lease_fixed_ip /usr/lib/python2.7/dist-packages/nova/network/manager.py:650 2012-02-27 17:37:33,489 WARNING nova.network.manager [be0218cd-335b-45d9-82dc-71cf06d1ac93 None None] IP |10.0.0.25| leased that isn't allocated 2012-02-27 17:37:34,394 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'f2179493-849d-4e80-95ac-9462415d9515', u'_context_read_deleted': False, u'args': {u'address': u'10.0.0.39'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-02-27T22:37:34.355962', u'_context_user_id': None, u'method': u'lease_fixed_ip', u'_context_remote_address': None} from (pid=1201) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:37:34,394 DEBUG nova.rpc [-] unpacked context: {'user_id': None, 'roles': [], 'timestamp': u'2012-02-27T22:37:34.355962', 'auth_token': None, 'msg_id': None, 'remote_address': None, 'strategy': u'noauth', 'is_admin': True, 'request_id': u'f2179493-849d-4e80-95ac-9462415d9515', 'project_id': None, 'read_deleted': False} from (pid=1201) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:37:34,395 DEBUG nova.network.manager [f2179493-849d-4e80-95ac-9462415d9515 None None] Leased IP |10.0.0.39| from (pid=1201) lease_fixed_ip /usr/lib/python2.7/dist-packages/nova/network/manager.py:650 2012-02-27 17:37:34,489 WARNING nova.network.manager [f2179493-849d-4e80-95ac-9462415d9515 None None] IP |10.0.0.39| leased that isn't allocated ############## nova-network log for other node 2012-02-27 17:35:27,262 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'611af1b6-bdcc-4779-83a3-1f7fcbdbad22', u'_context_read_deleted': False, u'args': {u'instance_id': 155, u'project_id': u'test'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'testproject', u'_context_timestamp': u'2012-02-27T22:30:33.977330', u'_context_user_id': u'tester', u'method': u'deallocate_for_instance', u'_context_remote_address': u'127.0.0.1'} from (pid=1187) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:27,262 DEBUG nova.rpc [-] unpacked context: {'user_id': u'tester', 'roles': [], 'timestamp': u'2012-02-27T22:30:33.977330', 'auth_token': None, 'msg_id': None, 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'611af1b6-bdcc-4779-83a3-1f7fcbdbad22', 'project_id': u'testproject', 'read_deleted': False} from (pid=1187) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:27,262 DEBUG nova.network.manager [611af1b6-bdcc-4779-83a3-1f7fcbdbad22 tester testproject] floating IP deallocation for instance |155| from (pid=1187) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251 2012-02-27 17:35:27,267 DEBUG nova.network.manager [611af1b6-bdcc-4779-83a3-1f7fcbdbad22 tester testproject] network deallocation for instance |155| from (pid=1187) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:465 2012-02-27 17:35:27,373 ERROR nova.rpc [2c483801-c696-4ede-8db9-a2f3e84f02f2 None None] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 275, in deallocate_for_instance (nova.rpc): TRACE: super(FloatingIP, self).deallocate_for_instance(context, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 468, in deallocate_for_instance (nova.rpc): TRACE: self.deallocate_fixed_ip(context, fixed_ip['address'], **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 641, in deallocate_fixed_ip (nova.rpc): TRACE: instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 404, in _do_trigger_security_group_members_refresh_for_instance (nova.rpc): TRACE: instance_ref = self.db.instance_get(admin_context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 500, in instance_get (nova.rpc): TRACE: return IMPL.instance_get(context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 119, in wrapper (nova.rpc): TRACE: return f(*args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1171, in instance_get (nova.rpc): TRACE: raise exception.InstanceNotFound(instance_id=instance_id) (nova.rpc): TRACE: InstanceNotFound: Instance 155 could not be found. (nova.rpc): TRACE: 2012-02-27 17:35:27,405 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'cd7ba21e-1213-4322-bd8f-2b1fadb48827', u'_context_read_deleted': False, u'args': {u'instance_id': 155, u'project_id': u'test'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'testproject', u'_context_timestamp': u'2012-02-27T22:27:15.267655', u'_context_user_id': u'tester', u'method': u'deallocate_for_instance', u'_context_remote_address': u'127.0.0.1'} from (pid=1187) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:27,405 DEBUG nova.rpc [-] unpacked context: {'user_id': u'tester', 'roles': [], 'timestamp': u'2012-02-27T22:27:15.267655', 'auth_token': None, 'msg_id': None, 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'cd7ba21e-1213-4322-bd8f-2b1fadb48827', 'project_id': u'testproject', 'read_deleted': False} from (pid=1187) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:27,406 DEBUG nova.rpc [cd7ba21e-1213-4322-bd8f-2b1fadb48827 tester testproject] received {u'_context_roles': [], u'_context_request_id': u'f38b1977-a2f5-4b1f-83df-e937977df1bc', u'_context_read_deleted': False, u'args': {u'instance_id': 200, u'project_id': u'test'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'test', u'_context_timestamp': u'2012-02-27T22:21:51.067532', u'_context_user_id': u'MedCloud', u'method': u'deallocate_for_instance', u'_context_remote_address': u'127.0.0.1'} from (pid=1187) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:27,406 DEBUG nova.rpc [cd7ba21e-1213-4322-bd8f-2b1fadb48827 tester testproject] unpacked context: {'user_id': u'MedCloud', 'roles': [], 'timestamp': u'2012-02-27T22:21:51.067532', 'auth_token': None, 'msg_id': None, 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'f38b1977-a2f5-4b1f-83df-e937977df1bc', 'project_id': u'test', 'read_deleted': False} from (pid=1187) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:27,407 DEBUG nova.network.manager [cd7ba21e-1213-4322-bd8f-2b1fadb48827 tester testproject] floating IP deallocation for instance |155| from (pid=1187) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251 2012-02-27 17:35:27,411 DEBUG nova.network.manager [cd7ba21e-1213-4322-bd8f-2b1fadb48827 tester testproject] network deallocation for instance |155| from (pid=1187) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:465 2012-02-27 17:35:27,490 ERROR nova.rpc [216e575e-00cf-4991-ae07-62732c175733 None None] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 275, in deallocate_for_instance (nova.rpc): TRACE: super(FloatingIP, self).deallocate_for_instance(context, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 468, in deallocate_for_instance (nova.rpc): TRACE: self.deallocate_fixed_ip(context, fixed_ip['address'], **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 641, in deallocate_fixed_ip (nova.rpc): TRACE: instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 404, in _do_trigger_security_group_members_refresh_for_instance (nova.rpc): TRACE: instance_ref = self.db.instance_get(admin_context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 500, in instance_get (nova.rpc): TRACE: return IMPL.instance_get(context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 119, in wrapper (nova.rpc): TRACE: return f(*args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1171, in instance_get (nova.rpc): TRACE: raise exception.InstanceNotFound(instance_id=instance_id) (nova.rpc): TRACE: InstanceNotFound: Instance 155 could not be found. (nova.rpc): TRACE: 2012-02-27 17:35:27,491 DEBUG nova.network.manager [f38b1977-a2f5-4b1f-83df-e937977df1bc MedCloud test] floating IP deallocation for instance |200| from (pid=1187) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251 2012-02-27 17:35:27,495 DEBUG nova.network.manager [f38b1977-a2f5-4b1f-83df-e937977df1bc MedCloud test] network deallocation for instance |200| from (pid=1187) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:465 2012-02-27 17:35:27,572 ERROR nova.rpc [db557f36-5a78-4fd3-960c-10c004de691f None None] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 275, in deallocate_for_instance (nova.rpc): TRACE: super(FloatingIP, self).deallocate_for_instance(context, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 468, in deallocate_for_instance (nova.rpc): TRACE: self.deallocate_fixed_ip(context, fixed_ip['address'], **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 641, in deallocate_fixed_ip (nova.rpc): TRACE: instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 404, in _do_trigger_security_group_members_refresh_for_instance (nova.rpc): TRACE: instance_ref = self.db.instance_get(admin_context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 500, in instance_get (nova.rpc): TRACE: return IMPL.instance_get(context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 119, in wrapper (nova.rpc): TRACE: return f(*args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1171, in instance_get (nova.rpc): TRACE: raise exception.InstanceNotFound(instance_id=instance_id) (nova.rpc): TRACE: InstanceNotFound: Instance 200 could not be found. (nova.rpc): TRACE: 2012-02-27 17:35:27,625 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'79fd8125-7a93-463e-b959-34ebdefd2445', u'_context_read_deleted': False, u'args': {u'instance_id': 155, u'project_id': u'test'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'testproject', u'_context_timestamp': u'2012-02-27T22:27:48.453097', u'_context_user_id': u'tester', u'method': u'deallocate_for_instance', u'_context_remote_address': u'127.0.0.1'} from (pid=1187) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:27,626 DEBUG nova.rpc [-] unpacked context: {'user_id': u'tester', 'roles': [], 'timestamp': u'2012-02-27T22:27:48.453097', 'auth_token': None, 'msg_id': None, 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'79fd8125-7a93-463e-b959-34ebdefd2445', 'project_id': u'testproject', 'read_deleted': False} from (pid=1187) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:27,626 DEBUG nova.network.manager [79fd8125-7a93-463e-b959-34ebdefd2445 tester testproject] floating IP deallocation for instance |155| from (pid=1187) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251 2012-02-27 17:35:27,630 DEBUG nova.network.manager [79fd8125-7a93-463e-b959-34ebdefd2445 tester testproject] network deallocation for instance |155| from (pid=1187) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:465 2012-02-27 17:35:27,707 ERROR nova.rpc [097aafc7-33cc-4e78-9dd3-38ec62ab07a2 None None] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 275, in deallocate_for_instance (nova.rpc): TRACE: super(FloatingIP, self).deallocate_for_instance(context, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 468, in deallocate_for_instance (nova.rpc): TRACE: self.deallocate_fixed_ip(context, fixed_ip['address'], **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 641, in deallocate_fixed_ip (nova.rpc): TRACE: instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 404, in _do_trigger_security_group_members_refresh_for_instance (nova.rpc): TRACE: instance_ref = self.db.instance_get(admin_context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 500, in instance_get (nova.rpc): TRACE: return IMPL.instance_get(context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 119, in wrapper (nova.rpc): TRACE: return f(*args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1171, in instance_get (nova.rpc): TRACE: raise exception.InstanceNotFound(instance_id=instance_id) (nova.rpc): TRACE: InstanceNotFound: Instance 155 could not be found. (nova.rpc): TRACE: 2012-02-27 17:35:34,861 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'121aa6aa-646a-48db-902f-e97c8d33ca7a', u'_context_read_deleted': False, u'args': {u'instance_id': 200, u'project_id': u'test'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'test', u'_context_timestamp': u'2012-02-27T22:23:27.638975', u'_context_user_id': u'MedCloud', u'method': u'deallocate_for_instance', u'_context_remote_address': u'127.0.0.1'} from (pid=1187) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:35:34,861 DEBUG nova.rpc [-] unpacked context: {'user_id': u'MedCloud', 'roles': [], 'timestamp': u'2012-02-27T22:23:27.638975', 'auth_token': None, 'msg_id': None, 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'121aa6aa-646a-48db-902f-e97c8d33ca7a', 'project_id': u'test', 'read_deleted': False} from (pid=1187) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:35:34,862 DEBUG nova.network.manager [121aa6aa-646a-48db-902f-e97c8d33ca7a MedCloud test] floating IP deallocation for instance |200| from (pid=1187) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:251 2012-02-27 17:35:34,867 DEBUG nova.network.manager [121aa6aa-646a-48db-902f-e97c8d33ca7a MedCloud test] network deallocation for instance |200| from (pid=1187) deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/manager.py:465 2012-02-27 17:35:34,943 ERROR nova.rpc [675fbd2e-e621-437b-a9e9-4cd4733ff2d8 None None] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data (nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 275, in deallocate_for_instance (nova.rpc): TRACE: super(FloatingIP, self).deallocate_for_instance(context, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 468, in deallocate_for_instance (nova.rpc): TRACE: self.deallocate_fixed_ip(context, fixed_ip['address'], **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 641, in deallocate_fixed_ip (nova.rpc): TRACE: instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 404, in _do_trigger_security_group_members_refresh_for_instance (nova.rpc): TRACE: instance_ref = self.db.instance_get(admin_context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 500, in instance_get (nova.rpc): TRACE: return IMPL.instance_get(context, instance_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 119, in wrapper (nova.rpc): TRACE: return f(*args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1171, in instance_get (nova.rpc): TRACE: raise exception.InstanceNotFound(instance_id=instance_id) (nova.rpc): TRACE: InstanceNotFound: Instance 200 could not be found. (nova.rpc): TRACE: 2012-02-27 17:37:55,074 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'8cbacabe-0bfc-4647-acb3-d0d811d31f57', u'_context_read_deleted': False, u'args': {u'address': u'10.0.0.34'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-02-27T22:37:55.035957', u'_context_user_id': None, u'method': u'lease_fixed_ip', u'_context_remote_address': None} from (pid=1187) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:37:55,074 DEBUG nova.rpc [-] unpacked context: {'user_id': None, 'roles': [], 'timestamp': u'2012-02-27T22:37:55.035957', 'auth_token': None, 'msg_id': None, 'remote_address': None, 'strategy': u'noauth', 'is_admin': True, 'request_id': u'8cbacabe-0bfc-4647-acb3-d0d811d31f57', 'project_id': None, 'read_deleted': False} from (pid=1187) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:37:55,075 DEBUG nova.network.manager [8cbacabe-0bfc-4647-acb3-d0d811d31f57 None None] Leased IP |10.0.0.34| from (pid=1187) lease_fixed_ip /usr/lib/python2.7/dist-packages/nova/network/manager.py:650 2012-02-27 17:38:40,310 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_context_request_id': u'c6a1aabe-af71-4974-931b-96006cc61aaf', u'_context_read_deleted': False, u'args': {u'address': u'10.0.0.34'}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-02-27T22:38:40.272518', u'_context_user_id': None, u'method': u'lease_fixed_ip', u'_context_remote_address': None} from (pid=1187) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:38:40,311 DEBUG nova.rpc [-] unpacked context: {'user_id': None, 'roles': [], 'timestamp': u'2012-02-27T22:38:40.272518', 'auth_token': None, 'msg_id': None, 'remote_address': None, 'strategy': u'noauth', 'is_admin': True, 'request_id': u'c6a1aabe-af71-4974-931b-96006cc61aaf', 'project_id': None, 'read_deleted': False} from (pid=1187) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:38:40,311 DEBUG nova.network.manager [c6a1aabe-af71-4974-931b-96006cc61aaf None None] Leased IP |10.0.0.34| from (pid=1187) lease_fixed_ip /usr/lib/python2.7/dist-packages/nova/network/manager.py:650 2012-02-27 17:39:03,806 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_msg_id': u'94128ac267cf4a248a5e30537ca799b9', u'_context_read_deleted': False, u'_context_request_id': u'7b993547-8821-49e2-a84a-6fa30a61ef6c', u'args': {u'instance_id': 16, u'host': u'xg01', u'instance_type_id': 5}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'test', u'_context_timestamp': u'2012-02-27T22:39:02.776814', u'_context_user_id': u'MedCloud', u'method': u'get_instance_nw_info', u'_context_remote_address': u'127.0.0.1'} from (pid=1187) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:39:03,806 DEBUG nova.rpc [-] unpacked context: {'user_id': u'MedCloud', 'roles': [], 'timestamp': u'2012-02-27T22:39:02.776814', 'auth_token': None, 'msg_id': u'94128ac267cf4a248a5e30537ca799b9', 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'7b993547-8821-49e2-a84a-6fa30a61ef6c', 'project_id': u'test', 'read_deleted': False} from (pid=1187) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:39:03,886 DEBUG nova.utils [-] Attempting to grab semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=1187) inner /usr/lib/python2.7/dist-packages/nova/utils.py:674 2012-02-27 17:39:15,839 DEBUG nova.rpc [-] received {u'_context_roles': [], u'_msg_id': u'0ad1f059c4fc4bc388d863bcde8a861d', u'_context_read_deleted': False, u'_context_request_id': u'd290b39f-5090-45d5-818c-1696f4639c20', u'args': {u'instance_id': 181, u'host': u'xg01', u'instance_type_id': 5}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'test', u'_context_timestamp': u'2012-02-27T22:39:09.556702', u'_context_user_id': u'MedCloud', u'method': u'get_instance_nw_info', u'_context_remote_address': u'127.0.0.1'} from (pid=1187) __call__ /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:600 2012-02-27 17:39:15,840 DEBUG nova.rpc [-] unpacked context: {'user_id': u'MedCloud', 'roles': [], 'timestamp': u'2012-02-27T22:39:09.556702', 'auth_token': None, 'msg_id': u'0ad1f059c4fc4bc388d863bcde8a861d', 'remote_address': u'127.0.0.1', 'strategy': u'noauth', 'is_admin': True, 'request_id': u'd290b39f-5090-45d5-818c-1696f4639c20', 'project_id': u'test', 'read_deleted': False} from (pid=1187) _unpack_context /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:646 2012-02-27 17:39:15,841 DEBUG nova.rpc [d290b39f-5090-45d5-818c-1696f4639c20 MedCloud test] received {u'_context_roles': [], u'_msg_id': u'b3abba2928084a8bacaa024768ac832c', u'_context_read_deleted': False, u'_context_request_id': u'b896ff29-1ea8-464d-99ee-fb42fb2eac82', u'args': {u'instance_id': 192, u'host': u'xg01', u'instance_type_id': 5}, u'_context_auth_token': None, u'_context_strategy': u'noauth', u'_context_is_admin': True, u'_context_project_id': u'test', u'_context