SM:3.1build24 HA: Sanity test report failures with "Lost connection to MySQL server at 'reading initial communication packet'

[sundarkh]

SM:3.1build24 HA
----------------------
sanity test case test_verify_object_logs, reports the following failure

Traceback (most recent call last):

  File "tcutils/wrappers.py", line 73, in wrapper

    result = function(self, *args, **kwargs)

  File "scripts/analytics/test_analytics.py", line 113, in test_verify_object_logs

    vn_obj=vn_obj, vm_name= vm1_name, project_name= self.inputs.project_name))

  File "/contrail-test/fixtures/vm_test.py", line 67, in __init__

    self.vnc_lib_h = self.connections.get_vnc_lib_h()

  File "common/connections.py", line 112, in get_vnc_lib_h

    project_id=self.get_project_id(),

  File "common/connections.py", line 81, in get_project_id

    auth = self.get_auth_h(project_name)

  File "common/connections.py", line 93, in get_auth_h

    project_name, self.inputs, self.logger)

  File "/contrail-test/fixtures/openstack.py", line 328, in __init__

    self.reauth()

  File "/contrail-test/fixtures/openstack.py", line 336, in reauth

    region_name=self.region_name)

  File "/contrail-test/fixtures/keystone_tests.py", line 24, in __init__

    insecure=insecure, region_name=region_name or 'RegionOne')

  File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 152, in __init__

    self.authenticate()

  File "/usr/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner

    return func(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 503, in authenticate

    resp = self.get_raw_token_from_identity_service(**kwargs)

  File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 196, in get_raw_token_from_identity_service

    _("Authorization Failed: %s") % e)

AuthorizationFailure: Authorization Failed: An unexpected error prevented the server from fulfilling your request: (OperationalError) (2013, "Lost connection to MySQL server at 'reading initial communication packet', system error: 110") None None (Disable debug mode to suppress these details.) (HTTP 500) (Request-ID: req-d951d8a1-a1f5-4e17-a91b-5155f5d643fc)

Development team aware of the issue.

Seen with kilo/liberty/mitaka

[sundarkh (sundar-kh)]

Seen with R3.0 build 60 Kilo

[sundarkh (sundar-kh)]

TestEvpnCasesVxlan.test_with_vxlan_encap_to_verify_l2_vm_file_trf_by_scp test case also affected due to this bug

[sundarkh (sundar-kh)]

few more sanity tests which fail due to this bug

TestECMPSanity.test_ecmp_svc_transparent_with_3_instance[ci_sanity_WIP,sanity]
TestECMPSanity.test_ecmp_svc_v2_transparent_with_3_instance

R3.0 build 60 Kilo HA, in ubuntu

[sundarkh (sundar-kh)]

Mainline Kilo 2743
-------------------

nodeg21 : mysql error.log

160929 3:45:24 [Note] WSREP: Member 2 (nodeg33) synced with group.
160929 3:46:01 [ERROR] Slave SQL: Error 'Operation CREATE USER failed for 'cmon'@'10.204.217.57'' on query. Default database: ''. Query: 'CREATE USER 'cmon'@'10.204.217.57' IDENTIFIED BY 'cmon'', Error_code: 1396
160929 3:46:01 [Warning] WSREP: RBR event 1 Query apply warning: 1, 496
160929 3:46:01 [Warning] WSREP: Ignoring error for TO isolated action: source: 9fe69bb7-8631-11e6-9425-ef3754594a7c version: 2 local: 0 state: APPLYING flags: 65 conn_id: 1153 trx_id: -1 seqnos (l: 508, g: 496, s: 495, d: 495, ts: 1475145961587347290)
160929 3:46:20 [ERROR] Slave SQL: Error 'Operation CREATE USER failed for 'cmon'@'10.204.217.73'' on query. Default database: ''. Query: 'CREATE USER 'cmon'@'10.204.217.73' IDENTIFIED BY 'cmon'', Error_code: 1396
160929 3:46:20 [Warning] WSREP: RBR event 1 Query apply warning: 1, 751
160929 3:46:20 [Warning] WSREP: Ignoring error for TO isolated action: source: c74c7ecc-8631-11e6-a3ad-8ae742b1b71f version: 2 local: 0 state: APPLYING flags: 65 conn_id: 661 trx_id: -1 seqnos (l: 763, g: 751, s: 750, d: 750, ts: 1475145980266684893)

root@nodeg21:~# ps -aux | grep mysql
root 16966 0.0 0.0 4440 660 ? S 03:43 0:00 /bin/sh /usr/bin/mysqld_safe --wsrep_cluster_address=gcomm://
mysql 18218 3.6 1.2 12551280 405460 ? Sl 03:43 43:42 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --wsrep-cluster-address=gcomm:// --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 --wsrep_start_position=00000000-0000-0000-0000-000000000000:-1
root 31047 0.0 0.0 10464 936 pts/17 S+ 23:31 0:00 grep --color=auto mysql
root@nodeg21:~#

[Dheeraj Gautam (dgautam)]

Updates so far:
This issue comes with Sanity and test cases ran in parallel. When running individual testcases or running sanity in serial, This issue doesn't appear.

Other possible cause was continuous restart of haproxy by hamon script. Restart of haproxy will cause all mysql and nova connections to reset and cause above reported issues. https://review.opencontrail.org/#/c/25626/ review fixes haproxy restart issue.

[Dheeraj Gautam (dgautam)]

marking as fix committed as cmon was restarting haproxy and possibly causing the mysql connections to go bad.

[Dheeraj Gautam (dgautam)]

marking won't fix in older releases.