cells traceback on every build

Bug #1240730 reported by Chris Behrens
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Chris Behrens

Bug Description

On build we get a message from child cells where instance['info_cache']['network_info'] is []... and not json-ified. We need to json-ify this before passing to the DB method.

This traceback is for internal mysqldb code, but would likely be some sort of problem against sqlalchemy also.

332407:2013-10-15 15:11:01.336 17434 ERROR nova.cells.messaging [req-32f45513-ca61-4f37-b1a8-eeeb6f5100f3 10085925 5853594] Error processing message locally: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '), `updated_at` = '2013-10-15 15:11:01.334702' WHERE `instance_uuid`='8da76d19-b' at line 1")
332408:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging Traceback (most recent call last):
332409:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/cells/messaging.py", line 204, in _process_locally
332410:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging resp_value = self.msg_runner._process_message_locally(self)
332411:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/cells/messaging.py", line 1220, in _process_message_locally
332412:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging return fn(message, **message.method_kwargs)
332413:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/cells/messaging.py", line 999, in instance_update_at_top
332414:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging message.ctxt, instance_uuid, info_cache)
332415:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/db/api.py", line 838, in instance_info_cache_update
332416:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging return IMPL.instance_info_cache_update(context, instance_uuid, values)
332417:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/openstack/common/db/api.py", line 103, in tpool_wrapper
332418:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging return self.__tpool.execute(attr, *args, **kwargs)
332419:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/eventlet/tpool.py", line 76, in tworker
332420:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging rv = meth(*args,**kwargs)
332421:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/db/mysqldb/api.py", line 58, in inner
332422:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging result = f(*args, **kwargs)
332423:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/db/mysqldb/api.py", line 171, in instance_info_cache_update
332424:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging info_cache = method(conn, ctxt, instance_uuid, values)
332425:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/db/mysqldb/instance_info_cache.py", line 44, in update
332426:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging rows = query.update(conn)
332427:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/db/mysqldb/sql.py", line 397, in update
332428:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging cursor = self._execute(conn, query, kwargs)
332429:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/db/mysqldb/sql.py", line 124, in _execute
332430:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging cursor = conn.execute(query, args=query_kwargs)
332431:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/db/mysqldb/connection.py", line 59, in inner
332432:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging return f(self, *args, **kwargs)
332433:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/nova/db/mysqldb/connection.py", line 243, in execute
332434:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging cursor.execute(query, args)
332435:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/MySQLdb/cursors.py", line 201, in execute
332436:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging self.errorhandler(self, exc, value)
332437:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging File "/opt/rackstack/472.13/nova/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
332438:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging raise errorclass, errorvalue
332439:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '), `updated_at` = '2013-10-15 15:11:01.334702' WHERE `instance_uuid`='8da76d19-b' at line 1")
332440:2013-10-15 15:11:01.336 17434 TRACE nova.cells.messaging

Chris Behrens (cbehrens)
tags: added: havana-rc-potential
Changed in nova:
assignee: nobody → Chris Behrens (cbehrens)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/52231

Thierry Carrez (ttx)
tags: added: havana-backport-potential
removed: havana-rc-potential
Chris Behrens (cbehrens)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/52231
Committed: http://github.com/openstack/nova/commit/509bd157d3afc11ccda2a3a53fc3770cee2da3f6
Submitter: Jenkins
Branch: master

commit 509bd157d3afc11ccda2a3a53fc3770cee2da3f6
Author: Chris Behrens <email address hidden>
Date: Wed Oct 16 22:13:29 2013 +0000

    Make cells info_cache updates more tolerant

    Allow lists or network models to be passed in for
    instance['info_cache']['network_info'] in cells updates. Ultimately
    these messages will become more consistent when fully converted to objects.

    Change-Id: Ifd46a3060418b3e2e4073c3b232937a18a225f99
    Closes-bug: 1240730

Changed in nova:
status: In Progress → Fix Committed
Changed in nova:
milestone: none → icehouse-1
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-1 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.