Deploy failed with Task[primary-database/1] Stopping the deployment process!

[Alexander Kurenyshev]

Found on CI [1]. Test Enable plugin, installed after create, and add nodes

Steps to reproduce:
1. Enable plugin
2. Deploy 2 new fuel_plugin_example_v4 node at cluster
(Nodes Under Test)

Plugin is here [2]

Expected result:
Deploy is successful

Actual result:
2017-03-22 03:34:41 INFO [17346] Casting message to Nailgun:
{"method"=>"transaction_resp",
 "args"=>
  {"task_uuid"=>"c94328b1-e8e1-4dc2-be6d-3a49fedc1a06",
   "status"=>"error",
   "progress"=>100,
   "error"=>
    "All nodes are finished. Failed tasks: Task[primary-database/1] Stopping the deployment process!"}}

At the puppet log:
2017-03-22 03:27:23 +0000 Puppet (err): Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -e CREATE USER 'root'@'localhost' IDENTIFIED BY PASSWORD '*637FF6C35614819270CBEA55BBDC733800A081E0'' retu
rned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
<--->
2017-03-22 03:27:23 +0000 /Package[xinetd] (notice): Dependency Mysql_user[root@localhost] has failures: true
2017-03-22 03:27:23 +0000 /Package[xinetd] (warning): Skipping because of failed dependencies
2017-03-22 03:27:23 +0000 /Stage[main]/Cluster::Galera_status/File[/etc/wsrepclustercheckrc] (notice): Dependency Mysql_user[root@localhost] has failures: true
2017-03-22 03:27:23 +0000 /Stage[main]/Cluster::Galera_status/File[/etc/wsrepclustercheckrc] (warning): Skipping because of failed dependencies
2017-03-22 03:27:23 +0000 /Stage[main]/Xinetd/File[/etc/xinetd.conf] (notice): Dependency Mysql_user[root@localhost] has failures: true
2017-03-22 03:27:23 +0000 /Stage[main]/Xinetd/File[/etc/xinetd.conf] (warning): Skipping because of failed dependencies
2017-03-22 03:27:23 +0000 /Stage[main]/Cluster::Galera_status/Augeas[galeracheck] (notice): Dependency Mysql_user[root@localhost] has failures: true
2017-03-22 03:27:23 +0000 /Stage[main]/Cluster::Galera_status/Augeas[galeracheck] (warning): Skipping because of failed dependencies
2017-03-22 03:27:23 +0000 /Stage[main]/Xinetd/File[/etc/xinetd.d] (notice): Dependency Mysql_user[root@localhost] has failures: true
2017-03-22 03:27:23 +0000 /Stage[main]/Xinetd/File[/etc/xinetd.d] (warning): Skipping because of failed dependencies
2017-03-22 03:27:23 +0000 /Stage[main]/Cluster::Galera_status/Xinetd::Service[galeracheck]/File[/etc/xinetd.d/galeracheck] (notice): Dependency Mysql_user[root@localhost] has failures: true
2017-03-22 03:27:23 +0000 /Stage[main]/Cluster::Galera_status/Xinetd::Service[galeracheck]/File[/etc/xinetd.d/galeracheck] (warning): Skipping because of failed dependencies
2017-03-22 03:27:23 +0000 /Stage[main]/Xinetd/Service[xinetd] (notice): Dependency Mysql_user[root@localhost] has failures: true
2017-03-22 03:27:23 +0000 /Stage[main]/Xinetd/Service[xinetd] (warning): Skipping because of failed dependencies
2017-03-22 03:27:23 +0000 /Stage[main]/Osnailyfacter::Database::Database_backend_wait/Osnailyfacter::Wait_for_backend[mysql]/Haproxy_backend_status[mysql] (notice): Dependency Mysql_user[root@localhost] has failures: true
2017-03-22 03:27:23 +0000 /Stage[main]/Osnailyfacter::Database::Database_backend_wait/Osnailyfacter::Wait_for_backend[mysql]/Haproxy_backend_status[mysql] (warning): Skipping because of failed dependencies
2017-03-22 03:27:24 +0000 Puppet (notice): Finished catalog run in 725.42 seconds
2017-03-22 03:27:31 +0000 Scope(Class[Osnailyfacter::Database::Database]) (notice): MODULAR: database/database.pp
2017-03-22 03:27:37 +0000 Puppet (notice): Compiled catalog for node-1.test.domain.local in environment production in 5.57 seconds
2017-03-22 03:27:39 +0000 /Stage[main]/Cluster::Mysql/Exec[create-init-file]/returns (notice): executed successfully
2017-03-22 03:28:58 +0000 /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure (notice): ensure changed 'stopped' to 'running'

Logs are here [3]

[1] https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.plugins.install_plugin_after_cluster_create/196/console
[2] https://product-ci.infra.mirantis.net/job/10.0.build-fuel-plugins/lastSuccessfulBuild/artifact/built_plugins/fuel_plugin_example_v4-4.0-4.0.0-1.noarch.rpm
[3] https://drive.google.com/file/d/0BzdDsIW-ymG2bkVzN2lHNWxmV2M/view?usp=sharing

[Oleksiy Molchanov (omolchanov)]

2017-03-22T03:00:42.700869+00:00 err: 2017-03-22 03:00:42 4408 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2017-03-22T03:00:42.700869+00:00 err: 2017-03-22 03:00:42 4408 [ERROR] Plugin 'InnoDB' init function returned error.
2017-03-22T03:00:42.700869+00:00 err: 2017-03-22 03:00:42 4408 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2017-03-22T03:00:42.700869+00:00 err: 2017-03-22 03:00:42 4408 [ERROR] Unknown/unsupported storage engine: innodb
2017-03-22T03:00:42.700869+00:00 err: 2017-03-22 03:00:42 4408 [ERROR] Aborting

[Vladimir Kuklin (vkuklin)]

Unknown storage engine innodb means that the mysql database package is corrupted almost surely. I suggest that this is some hardware bug. Let's see another reproduce. Incomplete until then.

[Nitesh (nitesh1990)]

Faced same issue on Fuel-9.2
HA Openstack having 3 Controller node.
-------------------------
2017-04-02 04:56:08 ERR /usr/bin/puppet:8:in `<main>'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:92:in `execute'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:146:in `run'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `run'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:496:in `exit_on_fail'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block in run'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:507:in `plugin_hook'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/application.rb:381:in `block (2 levels) in run'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:159:in `run_command'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:198:in `main'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:236:in `block in main'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/application/apply.rb:302:in `apply_catalog'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:133:in `run'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet.rb:246:in `override'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/context.rb:64:in `override'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:134:in `block in run'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `run_internal'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/configurer.rb:119:in `apply_catalog'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:160:in `benchmark'
2017-04-02 04:56:08 ERR /usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime'
2017-04-02 04:56:08 ERR /usr/lib/ruby/vendor_ruby/puppet/util.rb:161:in `block in benchmark'
--------------------------------------------------

[Alexander Kurenyshev (akurenyshev)]

@Nitesh,
please, provide any additional info like logs snapshot, etc if any.