Failed to connect to mysql during deploy

[Dmitry Kalashnik]

Steps to reproduce:
            1. Create a cluster
            2. Add 3 nodes with controller role
            3. Add 1 node with compute role
Failed step 4. Deploy the cluster
            5. Verify network configuration on controller
            6. Run OSTF
            7. Verify that the default hostname is applied on cluster nodes

Expected result:
Test PASS

Error:
2016-09-08 00:35:45 ERR (/Stage[main]/Cluster::Mysql/Exec[wait-for-sync]) mysql -uclustercheck -pZGZTXtfDUebRD5fwydiorNV5 -Nbe "show status like 'wsrep_local_state_comment'" | grep -q -e Synced && sleep 10 returned 1 instead of one of [0]
2016-09-08 00:35:45 ERR (/Stage[main]/Cluster::Mysql/Exec[wait-for-sync]) Failed to call refresh: mysql -uclustercheck -pZGZTXtfDUebRD5fwydiorNV5 -Nbe "show status like 'wsrep_local_state_comment'" | grep -q -e Synced && sleep 10 returned 1 instead of one of [0]
2016-09-08 00:35:45 NOTICE (/Stage[main]/Cluster::Mysql/Exec[wait-for-sync]/returns) ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

shotgun2 report: http://pastebin.com/LuEi2jzf
Snapshot #​233
Failed job: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.custom_hostname/52/

[Dmitry Kalashnik (dkalashnik)]

Also I've got the similar issue with custom 10

[Maksim Malchuk (mmalchuk)]

there is another issue with 10.0 on the node-1:

2016-09-08T09:55:53.122129+00:00 notice: (/Stage[main]/Cluster::Rabbitmq_fence/Exec[enable_corosync_notifyd]/returns) sed: can't read /etc/default/corosync-notifyd: No such file or directory
2016-09-08T09:55:53.124172+00:00 err: sed -i s/START=no/START=yes/ /etc/default/corosync-notifyd returned 2 instead of one of [0]

[Maksim Malchuk (mmalchuk)]

the actual fail looks strange, the Galera cluster didn't assembled and mysqld on the node-5 dies with:

2016-09-08T00:26:11.777287+00:00 err: 2016-09-08 00:26:11 24266 [ERROR] InnoDB: Only one log file found.
2016-09-08T00:26:11.777595+00:00 err: 2016-09-08 00:26:11 24266 [ERROR] Plugin 'InnoDB' init function returned error.
2016-09-08T00:26:11.777595+00:00 err: 2016-09-08 00:26:11 24266 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2016-09-08T00:26:11.777941+00:00 err: 2016-09-08 00:26:11 24266 [ERROR] Unknown/unsupported storage engine: innodb
2016-09-08T00:26:11.778213+00:00 err: 2016-09-08 00:26:11 24266 [ERROR] Aborting

and later tries to start again in the endless loop:

2016-09-08T00:26:20.890492+00:00 err: 2016-09-08 00:26:20 0 [Note] /usr/sbin/mysqld (mysqld 5.6.30-0~u14.04+mos1) starting as process 27861 ...
2016-09-08T00:31:22.095149+00:00 err: 2016-09-08 00:31:22 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation
 for more details).
2016-09-08T00:31:22.095583+00:00 err: 2016-09-08 00:31:22 0 [Note] /usr/sbin/mysqld (mysqld 5.6.30-0~u14.04+mos1) starting as process 12658 ...
2016-09-08T00:36:29.341820+00:00 err: 2016-09-08 00:36:29 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation
 for more details).
2016-09-08T00:36:29.341820+00:00 err: 2016-09-08 00:36:29 0 [Note] /usr/sbin/mysqld (mysqld 5.6.30-0~u14.04+mos1) starting as process 1203 ...
2016-09-08T00:41:29.872106+00:00 err: 2016-09-08 00:41:29 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation
 for more details).
2016-09-08T00:41:29.872423+00:00 err: 2016-09-08 00:41:29 0 [Note] /usr/sbin/mysqld (mysqld 5.6.30-0~u14.04+mos1) starting as process 17279 ...
2016-09-08T00:46:39.445401+00:00 err: 2016-09-08 00:46:39 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation
 for more details).
2016-09-08T00:46:39.445401+00:00 err: 2016-09-08 00:46:39 0 [Note] /usr/sbin/mysqld (mysqld 5.6.30-0~u14.04+mos1) starting as process 6055 ...
2016-09-08T00:51:45.528926+00:00 err: 2016-09-08 00:51:45 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation
 for more details).
2016-09-08T00:51:45.529210+00:00 err: 2016-09-08 00:51:45 0 [Note] /usr/sbin/mysqld (mysqld 5.6.30-0~u14.04+mos1) starting as process 3823 ...
2016-09-08T00:56:45.822749+00:00 err: 2016-09-08 00:56:45 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation
 for more details).
2016-09-08T00:56:45.823237+00:00 err: 2016-09-08 00:56:45 0 [Note] /usr/sbin/mysqld (mysqld 5.6.30-0~u14.04+mos1) starting as process 1024 ...
2016-09-08T01:01:46.155685+00:00 err: 2016-09-08 01:01:46 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_...

[Dmitry Kalashnik (dkalashnik)]

Ok, it is reproduced on another test: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.custom_hostname/53/testReport/junit/(root)/set_custom_hostname/set_custom_hostname/

2016-09-09 04:50:55 +0000 /Stage[main]/Cluster::Mysql/Exec[wait-for-sync]/returns (notice): ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysql
d/mysqld.sock' (2)
2016-09-09 04:50:55 +0000 /Stage[main]/Cluster::Mysql/Exec[wait-for-sync] (err): Failed to call refresh: mysql -uclustercheck -pwvLNkErLWthwVBcMOf39ViiX -Nbe "show status like 'w
srep_local_state_comment'" | grep -q -e Synced && sleep 10 returned 1 instead of one of [0]
2016-09-09 04:50:55 +0000 /Stage[main]/Cluster::Mysql/Exec[wait-for-sync] (err): mysql -uclustercheck -pwvLNkErLWthwVBcMOf39ViiX -Nbe "show status like 'wsrep_local_state_comment
'" | grep -q -e Synced && sleep 10 returned 1 instead of one of [0]

[Vladimir Khlyunev (vkhlyunev)]

Got repro on backup-restore jobs:
https://product-ci.infra.mirantis.net/view/upgrades/job/9.x.backup-restore.ubuntu.upgrade_ceph_ha_tests/lastFailedBuild/

[Vladimir Khlyunev (vkhlyunev)]

https://product-ci.infra.mirantis.net/view/upgrades/job/9.x.backup-restore.ubuntu.upgrade_plugin_tests/ this one too - its blocking verification of backup-restore feature (it was planned to close all qa activities over backup-restore today)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/367995

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/367996

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maksim Malchuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/367995
Reason: couldn't help with an issue

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maksim Malchuk (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/367996
Reason: couldn't help with an issue

[Maksim Malchuk (mmalchuk)]

summary for the failed issues: http://paste.openstack.org/show/572408/

[Maksim Malchuk (mmalchuk)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/367996

[Maksim Malchuk (mmalchuk)]

please remove 5.6_5.6.30 version packages from the proposed and snapsots

[OpenStack Infra (hudson-openstack)]

Change abandoned by Maksim Malchuk (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/367996

[Roman Vyalov (r0mikiam)]

mos-linux please revert the mysql packages from https://bugs.launchpad.net/mos/+bug/1578370

[Roman Vyalov (r0mikiam)]

the mysql packages were deleted http://mirror.seed-cz1.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-latest/pool/main/m/mysql-wsrep-5.6/

[Maksim Malchuk (mmalchuk)]

10.0 moved to the Invalid due to: https://bugs.launchpad.net/fuel/+bug/1623394

[Maksim Malchuk (mmalchuk)]

Andrey, JFYI:

New MySQL packages based on mysql-wsrep 5.6.33 are under review.

https://review.fuel-infra.org/#/c/26542/
https://review.fuel-infra.org/#/c/26544/

They contain all fixes we applied for E/// also the fix for latest CVE. IT passed BVT https://custom-ci.infra.mirantis.net/job/10.0.custom.ubuntu.bvt_2/316/

[Andrey Lavrentyev (alavrentyev)]

@Maksim, thanks for letting know.

Anyway I wasn't able to reproduce it on 9.1 snapshot #280, so moving it to Released...

[root@nailgun ~]# shotgun2 short-report
cat /etc/fuel_build_id:
 495
cat /etc/fuel_build_number:
 495
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0

UBUNTU_MIRROR_ID=ubuntu-2016-09-14-213640
CENTOS_MIRROR_ID=centos-7.2.1511-2016-05-31-083834
MOS_UBUNTU_MIRROR_ID=9.0-2016-09-17-040338
MOS_CENTOS_OS_MIRROR_ID=os-2016-06-23-135731
MOS_CENTOS_PROPOSED_MIRROR_ID=proposed-2016-09-17-040338
MOS_CENTOS_UPDATES_MIRROR_ID=updates-2016-06-23-135916
MOS_CENTOS_HOLDBACK_MIRROR_ID=holdback-2016-06-23-140047
MOS_CENTOS_HOTFIX_MIRROR_ID=hotfix-2016-09-16-084322
MOS_CENTOS_SECURITY_MIRROR_ID=security-2016-06-23-140002

[Dmitry Kalashnik (dkalashnik)]

Appeared again here: https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.cgroup_ha/76/

[Dmitry Kalashnik (dkalashnik)]

New snapshot

[Roman Vyalov (r0mikiam)]

remove infra team from assigned label.

[Nastya Urlapova (aurlapova)]

Test are green:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.custom_hostname/89/testReport/(root)/
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.cgroup_ha/89/testReport/

[Nastya Urlapova (aurlapova)]

Snapshot ​#367