The following bvt job was failed:
https://product-ci.infra.mirantis.net/job/9.x.main.ubuntu.bvt_2/386/console
The reason:
In downloaded snapshot node-1 has errors when neutron-db-manage command was failed with the following error:
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db [req-9845199d-0fff-41e6-b794-2cf6f570a0e6 - - - - -] Exception encountered during network rescheduling
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db Traceback (most recent call last):
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/neutron/db/agentschedulers_db.py", line 281, in remove_networks_from_down_agents
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db context, {'agent_type': [constants.AGENT_TYPE_DHCP]})
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/neutron/db/agents_db.py", line 271, in get_agents_db
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db return query.all()
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2588, in all
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db return list(self)
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db return self._execute_and_instances(context)
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2751, in _execute_and_instances
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db result = conn.execute(querycontext.statement, self._params)
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 914, in execute
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db return meth(self, multiparams, params)
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db return connection._execute_clauseelement(self, multiparams, params)
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db compiled_sql, distilled_params
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db context)
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1337, in _handle_dbapi_exception
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db util.raise_from_cause(newraise, exc_info)
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db reraise(type(exception), exception, tb=exc_tb, cause=cause)
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db context)
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 450, in do_execute
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db cursor.execute(statement, parameters)
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 219, in execute
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db self.errorhandler(self, exc, value)
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 38, in defaulterrorhandler
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db raise errorvalue
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db OperationalError: (_mysql_exceptions.OperationalError) (1054, "Unknown column 'agents.availability_zone' in 'field list'") [SQL: u'SELECT agents.id AS agents_id, agents.agent_type AS agents_agent_type, agents.`binary` AS agents_binary, agents.topic AS agents_topic, agents.host AS agents_host, agents.availability_zone AS agents_availability_zone, agents.admin_state_up AS agents_admin_state_up, agents.created_at AS agents_created_at, agents.started_at AS agents_started_at, agents.heartbeat_timestamp AS agents_heartbeat_timestamp, agents.description AS agents_description, agents.configurations AS agents_configurations, agents.resource_versions AS agents_resource_versions, agents.`load` AS agents_load \nFROM agents \nWHERE agents.agent_type IN (%s)'] [parameters: ('DHCP agent',)]
2016-10-14 07:30:09.747 13811 ERROR neutron.db.agentschedulers_db
Before that time "07:30" in pacemaker.log it was observed some outage of mysql service with the following errors:
http://paste.openstack.org/show/585731/
So during this outage neutron database was corrupted and all migrations were failed during the neutron-db-manage
Medium is because very rare and strange issue, never seen this before.