Mysql can't start properly, "InnoDB" init function returned error.

[Alexey. Kalashnikov]

Swarm test "Install plugin after cluster create" failed with following message:
http://paste.openstack.org/show/587114/
Cluster is not deployed: some nodes are in the Error state
----------------------------------------------------------
This is what I dig out from logs on server

1. error in astute.log:
"status"=>"error",
      "error_type"=>"deploy",
      "task_status"=>"error",
      "error_msg"=>"Task database failed on node 3"

2. cut from syslog log:
<27>Oct 25 22:44:09 node-3 ocf-mysql-wss: ERROR: p_mysqld: proc_kill(): cannot find any processes matching the mysqld.*/var/lib/mysql!
<27>Oct 25 22:44:09 node-3 ocf-mysql-wss: ERROR: p_mysqld: proc_kill(): cannot find any processes matching the mysqld.*/var/lib/mysql!
<27>Oct 25 22:44:09 node-3 ocf-mysql-wss: ERROR: p_mysqld: proc_stop(): ERROR: could not stop mysqld.*/var/lib/mysql
<27>Oct 25 22:44:12 node-3 ocf-mysql-wss: ERROR: p_mysqld: mysql_status(): PIDFile /var/run/resource-agents/mysql-wss/mysql-wss.pid of MySQL server not found. Sleeping for 2 seconds. 0 retries left
<27>Oct 25 22:44:14 node-3 ocf-mysql-wss: ERROR: p_mysqld: mysql_status(): MySQL is not running
<27>Oct 25 22:49:10 node-3 ocf-mysql-wss: ERROR: p_mysqld: proc_kill(): cannot find any processes matching the mysqld.*/var/lib/mysql!
<27>Oct 25 22:49:10 node-3 ocf-mysql-wss: ERROR: p_mysqld: proc_kill(): cannot find any processes matching the mysqld.*/var/lib/mysql!
<27>Oct 25 22:49:10 node-3 ocf-mysql-wss: ERROR: p_mysqld: proc_stop(): ERROR: could not stop mysqld.*/var/lib/mysql
<27>Oct 25 22:54:28 node-3 ocf-mysql-wss: ERROR: p_mysqld: mysql_status(): PIDFile /var/run/resource-agents/mysql-wss/mysql-wss.pid of MySQL server not found. Sleeping for 2 seconds. 0 retries left
<27>Oct 25 22:54:30 node-3 ocf-mysql-wss: ERROR: p_mysqld: mysql_status(): MySQL is not running

3 and cut mysql logs:
2016-10-25 22:49:26 1839 [ERROR] Plugin 'InnoDB' init function returned error.
2016-10-25 22:49:26 1839 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2016-10-25 22:49:26 1839 [ERROR] Unknown/unsupported storage engine: innodb
2016-10-25 22:49:26 1839 [ERROR] Aborting

diagnostic snapshot:
https://drive.google.com/open?id=0B0EB6QSDWt2vUHU0NUN1TFZacmM

Additional info:
console output of test:
https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.plugins.install_plugin_after_cluster_create/112/consoleFull

[Sergey Novikov (snovikov)]

the same issue for https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.network_templates/126/testReport/(root)/deploy_ceph_net_tmpl/deploy_ceph_net_tmpl/