[swarm 9.2]ceph_partitions_repetitive_cold_restart: Cannot connect to the Docker daemon

[ElenaRossokhina]

Detailed bug description:
found on https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.repetitive_restart/169
Steps to reproduce:
1. Revert snapshot 'prepare_load_ceph_ha'
2. Wait until MySQL Galera is UP on some controller
3. Check Ceph status
4. Run ostf
5. Fill ceph partitions on all nodes up to 30%
6. Check Ceph status
7. Disable UMM
8. Run RALLY (fails here)
9. 100 times repetitive reboot:
10. Cold restart of all nodes
11. Wait for HA services ready
12. Wait until MySQL Galera is UP on some controller
13. Run ostf

Error on step 8: Docker image for Rally not found!

It occurs due to docker is dead on master node:
[root@nailgun ~]# service docker status
Redirecting to /bin/systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: http://docs.docker.com

Jan 12 04:10:31 nailgun.test.domain.local systemd[1]: Collecting docker.service

sys_test.log: 2017-01-12 04:10:27 - DEBUG ssh_client.py:731 -- docker pull docker.io/rallyforge/rally execution results: Exit code: 1
2017-01-12 04:10:27 - DEBUG rally.py:58 -- ExecResult(
 cmd=docker pull docker.io/rallyforge/rally,
  stdout=
'Using default tag: latest',
 stderr=
'Cannot connect to the Docker daemon. Is the docker daemon running on this host?',
 exit_code=1
)

[ElenaRossokhina (esolomina)]

diagnostic snapshot https://drive.google.com/open?id=0B2ag_Bf-ShtTeGhVMEtXSENiQjQ

[Dmitry Pyzhov (dpyzhov)]

Rally docker image is not part of Fuel distribution. Looks like issue with test configuration.

[Alexander Kurenyshev (akurenyshev)]

It's not a configuration problem: the docker rally image downloading url is configured here [1]:
docker.io/rallyforge/rally

But like it was said in description the docker.service is not running at the master node.
After I manually have started the docker I was able to pull image like test does:

[root@nailgun ~]# docker pull docker.io/rallyforge/rally
Using default tag: latest
Trying to pull repository docker.io/rallyforge/rally ...
latest: Pulling from docker.io/rallyforge/rally
f2e9c02f3e2d: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:5179baf1017c53f89390873db89c63e138e40bbd76daf800a63998239fe39345
Status: Downloaded newer image for docker.io/rallyforge/rally:latest
[root@nailgun ~]#
[root@nailgun ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/rallyforge/rally latest 04d16d90b65f 43 hours ago 720.9 MB

The root cause of problem is docker service doesn't start automatically after master node restart.
If we perform `systemctl enable docker.service` this will enables autostart of docker.

[1] https://github.com/openstack/fuel-qa/blob/8c798b4f370ea1c1e6c9a6f6a6dd169f825f7e8f/fuelweb_test/settings.py#L702

[Fuel Devops McRobotson (fuel-devops-robot)]

Related fix proposed to branch: stable/mitaka
Change author: Vladimir Jigulin <email address hidden>
Review: https://review.fuel-infra.org/37237

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/37237
Submitter: Dmitry Burmistrov <email address hidden>
Branch: stable/mitaka

Commit: f134aa9114cb551b02194aa2e8df85954b31e221
Author: Vladimir Jigulin <email address hidden>
Date: Thu Nov 30 08:19:45 2017

Start docker service on master node when we want use rally

Change-Id: Ide62a654737348cc44e1be4422fbb0fd200c030a
Related-Bug: #1656047