Deployment has failed on task[openstack-network-networks/3] with mysql exception

[Tatyana Kuterina]

Detailed bug description:
Found on CI:

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.bonding_ha/66/testReport/(root)/deploy_bonding_neutron_tun/deploy_bonding_neutron_tun/

Test: Deploy cluster with active-backup bonding and Neutron VXLAN
Test Group: deploy_bonding_neutron_tun

Steps to reproduce:
    Scenario:
        1. Create cluster
        2. Add 3 nodes with controller role
        3. Add 1 node with compute role and 1 node with cinder role
        4. Setup bonding for all interfaces (including admin interface
           bonding)
        5. Run network verification
        6. Deploy the cluster

Expected results:
   Deploy is finished without errors

Actual result:
   Deployment has failed. All nodes are finished. Failed tasks: Task[openstack-network-networks/3] Stopping the deployment process!

in neutron-server.log:
oslo_messaging.rpc.dispatcher [req-534b8934-0baa-4bfd-9476-cd0a4b5f1ef0 - - - - -] Exception during message handling: (_mysql_exceptions.OperationalError) (1054, "Unknown column 'agents.availability_zone' in 'field list'")

in nova-cert.log:
nova.servicegroup.drivers.db [-] Unexpected error while reporting service status

Description of the environment:

9.1 snapshot #284
https://drive.google.com/a/mirantis.com/file/d/0Bz15vbpS5ZPNSF84Nnk5NU9oV3c/view?usp=sharing

[Oleksiy Molchanov (omolchanov)]

Neutron team, can you check? Seems that mysql was okay.

[Maksim Malchuk (mmalchuk)]

the error:

2016-09-19T23:34:35.311838+00:00 err: 2016-09-19 23:34:35.311 28960 ERROR oslo_messaging.rpc.dispatcher [req-534b8934-0baa-4bfd-9476-cd0a4b5f1ef0 - - - - -] Exception during message handling: (_mysql_exceptions.OperationalError) (1054, "Unknown column 'agents.availability_zone' in 'field list'") [SQL: u'SELEC
T agents.id AS agents_id, agents.agent_type AS agents_agent_type, agents.`binary` AS agents_binary, agents.topic AS agents_topic, agents.host AS agents_host, agents.availability_zone AS agents_availability_zone, agents.admin_state_up AS agents_admin_state_up, agents.created_at AS agents_created_at, agents.star
ted_at AS agents_started_at, agents.heartbeat_timestamp AS agents_heartbeat_timestamp, agents.description AS agents_description, agents.configurations AS agents_configurations, agents.resource_versions AS agents_resource_versions, agents.`load` AS agents_load \nFROM agents \nWHERE agents.agent_type = %s AND ag
ents.host = %s'] [parameters: ('Open vSwitch agent', 'node-3.test.domain.local')]

caused by:

2016-09-19T23:33:49.358909+00:00 err: ERROR: p_mysqld: check_if_galera_pc(): But I'm running a new cluster, PID:17842, this is a split-brain!

MySQL was shut down:

2016-09-19T23:33:49.659672+00:00 err: 2016-09-19 23:33:49 17842 [Note] /usr/sbin/mysqld: Normal shutdown

and successfully restarted by pacemaker.