Glance Lost connection to MySQL server at 'reading aut horization packet (System test deploy_bonding_ha_balance_slb)

[Egor Kotko]

{"build_id": "2014-11-11_20-08-59", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "33", "auth_required": true, "api": "1.0", "nailgun_sha": "bbc9dfe78a0c33040dcd16de9a40a3491788719c", "production": "docker", "fuelmain_sha": "88d4289e88e4bd88a3cabaf15a11ae8fc9ded53f", "astute_sha": "702af3db6f5bca92525bc8322d7d5d7675ec857e", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.1-5.1.1": {"VERSION": {"build_id": "2014-11-11_20-08-59", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "33", "api": "1.0", "nailgun_sha": "bbc9dfe78a0c33040dcd16de9a40a3491788719c", "production": "docker", "fuelmain_sha": "88d4289e88e4bd88a3cabaf15a11ae8fc9ded53f", "astute_sha": "702af3db6f5bca92525bc8322d7d5d7675ec857e", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "e5b3de834a400d98d8c6ba416249832a0c16076c"}}}, "fuellib_sha": "e5b3de834a400d98d8c6ba416249832a0c16076c"}

Scenario:
            1. Create cluster
            2. Add 3 nodes with controller role
            3. Add 2 node with compute role
            4. Setup bonding for all interfaces
            4. Deploy the cluster
            5. Run network verification
            6. Run OSTF

Glance log contains:
http://paste.openstack.org/show/132328/

Errors in puppet log on the second controller:
http://paste.openstack.org/show/132302/

http://jenkins-product.srt.mirantis.net:8080/view/5.1_swarm/job/5.1_fuelmain.system_test.ubuntu.bonding_ha/44/testReport/%28root%29/deploy_bonding_ha_balance_slb/deploy_bonding_ha_balance_slb/

[Bogdan Dobrelya (bogdando)]

See https://bugs.launchpad.net/mos/+bug/1352931/comments/3

[Bogdan Dobrelya (bogdando)]

The root cause is how Openstack handles 2013 mysql connectivity errors and that is why this bug is marked as a dup of known Oslo.db issue.

And the cause of 2013 errors, according to logs, are network flapping outlasted more than 1 hour, from 2014-11-12T04:56:37 to 2014-11-12T06:10:34 (just use 'Server.*DOWN|EXIT.*=1' patterns in logs)