several tasks fail permanently on bvt/smoke

[Aleksey Kasatkin]

Three ISOs in row fail on bvt/smoke with the following:

smokes:

Deployment has failed. All nodes are finished. Failed tasks: Task[cinder-db/3], Task[heat-db/3], Task[primary-keystone/3], Task[nova-db/3], Task[glance-db/3], Task[neutron-db/3] Stopping the deployment process!

Deployment has failed. All nodes are finished. Failed tasks: Task[cinder-db/3], Task[nova-db/3], Task[primary-keystone/3], Task[glance-db/3], Task[neutron-db/3] Stopping the deployment process!

Deployment has failed. All nodes are finished. Failed tasks: Task[cinder-db/1], Task[nova-db/1], Task[heat-db/1], Task[primary-keystone/1], Task[glance-db/1], Task[neutron-db/1] Stopping the deployment

bvts:

Deployment has failed. All nodes are finished. Failed tasks: Task[primary-keystone/1] Stopping the deployment process!

These are for 10.0.main.ubuntu.smoke_neutron/158-160 and for 10.0.main.ubuntu.bvt_2/158-160,
May, 5-6, 2016.

[Aleksey Kasatkin (alekseyk-ru)]

bvt logs contain:

node-2.test.domain.local 2016-05-06T10:29:10.857321 err: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
node-2.test.domain.local 2016-05-06T10:29:10.857433 err: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql --defaults-extra-file=/root/.my.cnf -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

...

node-1.test.domain.local 2016-05-06T10:58:59.827406 err: (/Stage[main]/Osnailyfacter::Wait_for_keystone_backends/Osnailyfacter::Wait_for_backend[keystone]/Haproxy_backend_status[keystone-admin]/ensure) change from down to up failed: Timeout waiting for HAProxy backend: 'keystone-2' status to become: 'up' after 1200 seconds!
node-1.test.domain.local 2016-05-06T10:58:59.827803 err: (/Stage[main]/Osnailyfacter::Wait_for_keystone_backends/Osnailyfacter::Wait_for_backend[keystone]/Haproxy_backend_status[keystone-admin]/ensure) change from down to up failed: Timeout waiting for HAProxy backend: 'keystone-2' status to become: 'up' after 1200 seconds!

[Aleksey Kasatkin (alekseyk-ru)]

same messages regarding keystone are in smoke logs.