Data Too Long for configurations column of neutron agents table

[Esha Seth]

When deploying with several networks the configurations column in table neutron_agents can run out of space causing the below error.

09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters [req-817b01ad-1b7c-47bc-b3c9-81ad798a6eda - - - - -] DBAPIError exception wrapped from (_mysql_exceptions.DataError) (1406, "Data too long for column 'configurations' at row 1") [SQL: u'UPDATE agents SET started_at=%s, heartbeat_timestamp=%s, configurations=%s WHERE agents.id = %s'] [parameters: (datetime.datetime(2016, 9, 6, 14, 3, 43, 663671), datetime.datetime(2016, 9, 6, 14, 3, 43, 663671), '{"bridge_mappings": {"ETHERNET0-VLAN488": "9425b666-6e94-3783-ba02-0bdbd0a5175a", "ETHERNET0-VLAN489": "9425b666-6e94-3783-ba02-0bdbd0a5175a", ..., "ETHERNET0-VLAN533": "9425b666-6e94-3783-ba02-0bdbd0a5175a"}, "devices": 0}', '44d71991-b258-4b23-8cdb-f65e5273dbd9')]
2016-09-06 09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2016-09-06 09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-09-06 09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters context)
2016-09-06 09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-09-06 09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
2016-09-06 09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2016-09-06 09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters self.errorhandler(self, exc, value)
2016-09-06 09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2016-09-06 09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters raise errorclass, errorvalue
2016-09-06 09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters DataError: (1406, "Data too long for column 'configurations' at row 1")
2016-09-06 09:03:43.791 970 ERROR oslo_db.sqlalchemy.exc_filters

2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher ectxt.value = e.inner_exc
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 137, in wrapper
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 472, in report_state
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher agent_status = self.plugin.create_or_update_agent(context, agent_state)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 386, in create_or_update_agent
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher return self._create_or_update_agent(context, agent)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/neutron/db/agents_db.py", line 380, in _create_or_update_agent
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher greenthread.sleep(0)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 502, in __exit__
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher self.rollback()
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 499, in __exit__
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher self.commit()
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher self._prepare_impl()
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher self.session.flush()
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2027, in flush
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher self._flush(objects)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2145, in _flush
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher transaction.rollback(_capture_exception=True)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2109, in _flush
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher flush_context.execute()
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in execute
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher rec.execute(self)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in execute
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher uow
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 170, in save_obj
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher mapper, table, update)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 706, in _emit_update_statements
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher execute(statement, multiparams)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher return meth(self, multiparams, params)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher return connection._execute_clauseelement(self, multiparams, params)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher compiled_sql, distilled_params
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher context)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher util.raise_from_cause(newraise, exc_info)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher reraise(type(exception), exception, tb=exc_tb)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher context)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher cursor.execute(statement, parameters)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher self.errorhandler(self, exc, value)
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2016-09-06 09:03:43.935 970 ERROR oslo_messaging.rpc.dispatcher raise errorclass, errorvalue

[Assaf Muller (amuller)]

Can you say what database do you use (MySQL, Postgress, Galera?) and what version?

[Reedip (reedip-banerjee-deactivatedaccount)]

Hey Esha,
Can you please tell us :
a) the number of networks launched ( to be sure of the condition you executed which hit this bug )
b) the version of neutron you are currently working on ?

[Reedip (reedip-banerjee-deactivatedaccount)]

amuller : seems to be mysql , isnt it ?
>>DBAPIError exception wrapped from (_mysql_exceptions.DataError) .....

[Reedip (reedip-banerjee-deactivatedaccount)]

amuller: http://dev.mysql.com/doc/refman/5.7/en/column-count-limit.html ??

[Drew Thorstensen (thorst)]

Esha - keep me honest. But I thought your issue per our discussion was that your environment had a very long bridge mapping. And the bridge mappings you had in the environment was larger than 4095, and thus you hit this issue. I didn't think it was actually tied to a deploy of a server itself...but rather when the agent reported status it failed to update the status because of the very long bridge mappings entry.

[Hirofumi Ichihara (ichihara-hirofumi)]

Could you reply some comments above?

[Esha Seth (eshaseth)]

Drew, yes the issue was due to very long bridge_mappings kept in configurations column which were exceeding the limit of the column. It is not related to deploy otherwise. I am cancelling this defect as invalid.