Deployment of ha cluster fail on execution keystone-manage sync db

[Tatyanka]

Deploy cluster in HA mode with Neutron GRE

Scenario:
1. Create cluster
2. Add 3 nodes with controller role
3. Add 2 nodes with compute role
4. Deploy the cluster
5. Run network verification
6. Run OSTF

Expected:
Deployment is ready

Actual:
Deployment fail with error in puppet.log (node-1)
2016-03-18T00:55:45.089285+00:00 err: (/Stage[main]/Keystone::Db::Sync/Exec[keystone-manage db_sync]) Failed to call refresh: keystone-manage db_sync returned 1 instead of one of [0]

in keystone-manage.log:
2016-03-18T00:55:45.013957+00:00 crit: 2016-03-18 00:55:45.008 5045 CRITICAL keystone [-] OperationalError: (_mysql_exceptions.OperationalError) (1047, 'Unknown command') [SQL: 'SELECT DATABASE()']
2016-03-18 00:55:45.008 5045 ERROR keystone Traceback (most recent call last):
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/bin/keystone-manage", line 10, in <module>
2016-03-18 00:55:45.008 5045 ERROR keystone sys.exit(main())
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/cmd/manage.py", line 47, in main
2016-03-18 00:55:45.008 5045 ERROR keystone cli.main(argv=sys.argv, config_files=config_files)
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/cmd/cli.py", line 992, in main
2016-03-18 00:55:45.008 5045 ERROR keystone CONF.command.cmd_class.main()
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/cmd/cli.py", line 371, in main
2016-03-18 00:55:45.008 5045 ERROR keystone migration_helpers.sync_database_to_version(extension, version)
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/common/sql/migration_helpers.py", line 210, in sync_database_to_version
2016-03-18 00:55:45.008 5045 ERROR keystone _sync_common_repo(version)
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/common/sql/migration_helpers.py", line 132, in _sync_common_repo
2016-03-18 00:55:45.008 5045 ERROR keystone with sql.session_for_write() as session:
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2016-03-18 00:55:45.008 5045 ERROR keystone return self.gen.next()
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 759, in _transaction_scope
2016-03-18 00:55:45.008 5045 ERROR keystone allow_async=self._allow_async) as resource:
2016-03-18 00:55:45.0
2016-03-18T00:55:45.014082+00:00 crit: 2016-03-18 00:55:45.008 5045 CRITICAL keystone [-] OperationalError: (_mysql_exceptions.OperationalError) (1047, 'Unknown command') [SQL: 'SELECT DATABASE()']
2016-03-18 00:55:45.008 5045 ERROR keystone Traceback (most recent call last):
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/bin/keystone-manage", line 10, in <module>
2016-03-18 00:55:45.008 5045 ERROR keystone sys.exit(main())
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/cmd/manage.py", line 47, in main
2016-03-18 00:55:45.008 5045 ERROR keystone cli.main(argv=sys.argv, config_files=config_files)
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/cmd/cli.py", line 992, in main
2016-03-18 00:55:45.008 5045 ERROR keystone CONF.command.cmd_class.main()
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/cmd/cli.py", line 371, in main
2016-03-18 00:55:45.008 5045 ERROR keystone migration_helpers.sync_database_to_version(extension, version)
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/common/sql/migration_helpers.py", line 210, in sync_database_to_version
2016-03-18 00:55:45.008 5045 ERROR keystone _sync_common_repo(version)
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/keystone/common/sql/migration_helpers.py", line 132, in _sync_common_repo
2016-03-18 00:55:45.008 5045 ERROR keystone with sql.session_for_write() as session:
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2016-03-18 00:55:45.008 5045 ERROR keystone return self.gen.next()
2016-03-18 00:55:45.008 5045 ERROR keystone File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 759, in _transaction_scope
2016-03-18 00:55:45.008 5045 ERROR keystone allow_async=self._allow_async) as resource:

iso: 76 package verison:
http://paste.openstack.org/show/491068/

[Tatyanka (tatyana-leontovich)]

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ha_neutron/50/testReport/junit/(root)/deploy_neutron_gre_ha/deploy_neutron_gre_ha/

[Tatyanka (tatyana-leontovich)]

+ occurrence 76 https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.network_templates/50/testReport/(root)/network_config_consistency_on_reboot/network_config_consistency_on_reboot/

[Tatyanka (tatyana-leontovich)]

+ occurrence https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ha_destructive_ceph_neutron/50/

so look slike it is swarm blocker for now

[Tatyanka (tatyana-leontovich)]

+ occurrence https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ha_neutron_destructive/51/testReport/junit/(root)/prepare_ha_neutron/prepare_ha_neutron/

[Tatyanka (tatyana-leontovich)]

+ occurrence https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.jumbo_frames/48/testReport/junit/(root)/jumbo_frames_neutron_vlan/jumbo_frames_neutron_vlan/

[Tatyanka (tatyana-leontovich)]

Set to critical according to it fails too may cases

[Dmitry Bilunov (dbilunov)]

This problem does not seem to be related with #1541338
Just to be sure let's wait for https://review.openstack.org/#/c/294173/ to be merged and then try to reproduce this issue.

[Dmitry Bilunov (dbilunov)]

Error 1047 is generated by galera in case if replication process has not been yet completed or the lag is too high (triggered by flow control mechanism).

[Ivan Berezovskiy (iberezovskiy)]

I set incomplete status till we get a reproduce after mysql patches landed.

[Bug Checker Bot (esikachev-l)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Tatyanka (tatyana-leontovich)]

reproduced https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.ha_destructive_ceph_neutron/54/testReport/(root)/prepare_ha_ceph_neutron/prepare_ha_ceph_neutron/

keystone sync db failed by timeout - conenction to mysql fail:
http://paste.openstack.org/show/491472/

http://paste.openstack.org/show/491473/

[Ivan Berezovskiy (iberezovskiy)]

Mysql was dead at the moment (23:01) of keystone db_sync. Logs from pacemaker: http://paste.openstack.org/show/491476/.
Fuel-library folks, please take a look.

[Dmitry Bilunov (dbilunov)]

% tail node-*/var/log/mysql*/error.log
==> node-3/var/log/mysql/error.log <==
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f2f7584b47d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f2f580009f0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

==> node-4/var/log/mysql/error.log <==
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f85a4f9347d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f85880009f0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

==> node-5/var/log/mysql/error.log <==
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ffbb03d147d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7ffb880009f0): is an invalid pointer
Connection ID (thread ID): 1
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.