Docker containers are not restarted by `dockerctl restart` command after setup

[Alexandr Kostrikov]

We got problem with enablement of advanced mode in fuel-qa on ci
https://product-ci.infra.mirantis.net/job/8.0.system_test.ubuntu.reduced_footprint/127/console

Expected:
We are setting advanced feature group at https://github.com/openstack/fuel-qa/blob/master/fuelweb_test/helpers/checkers.py#L361

def enable_feature_group(env, group):
    fuel_settings = env.admin_actions.get_fuel_settings()
    fuel_settings["FEATURE_GROUPS"].append(group)
    env.admin_actions.save_fuel_settings(fuel_settings)
    env.docker_actions.restart_container("nailgun")

    def check_api_available():
        try:
            env.fuel_web.client.get_api_version()
        except (urllib2.HTTPError, urllib2.URLError):
            return False
        return True

    wait(check_api_available, interval=10, timeout=60 * 15)
    wait(lambda: group in
         env.fuel_web.client.get_api_version()["feature_groups"],
         interval=10, timeout=60 * 5)

The calls are:
checkers.enable_feature_group(self.env, "advanced");

wait(lambda: group in
     env.fuel_web.client.get_api_version()["feature_groups"];

Astute.yaml to update group is at http://paste.openstack.org/show/485847/

Actual result:
Check is spinning in circles with message and fails because there is no advanced group in test:
2016-02-02 22:09:15,772 - DEBUG __init__.py:59 -- Done: get_api_version with result: {u'release': u'8.0', u'api': u'1', u'openstack_version': u'liberty-8.0', u'feature_groups': [u'mirantis'], u'auth_required': True};

[Artem Panchenko (apanchenko-8)]

According to the tests logs Nailgun container was restarted after adding of new feature group:

2016-02-02 22:04:14,851 - DEBUG ssh_manager.py:104 -- SSH_MANAGER:Return existed connection for 10.109.30.2:22
2016-02-02 22:04:14,851 - DEBUG ssh_manager.py:105 -- SSH_MANAGER: Connections {('10.109.30.2', 22): <devops.helpers.helpers.SSHClient object at 0x7f00d9264f10>}
2016-02-02 22:04:14,851 - DEBUG helpers.py:335 -- Executing command: 'cd ~'
2016-02-02 22:04:14,931 - DEBUG helpers.py:335 -- Executing command: 'dockerctl restart nailgun'

but Nailgun wasn't restarted:

fgrep 'Starting uWSGI' /var/log/docker-logs/nailgun/app.log
*** Starting uWSGI 2.0.11.1 (64bit) on [Tue Feb 2 21:16:40 2016] ***
fgrep -c 'Starting uWSGI' /var/log/docker-logs/nailgun/app.log
1

I tried to execute `dockerctl restart nailgun` command manually few times and it worked fine.

[Artem Panchenko (apanchenko-8)]

I've found the root cause - systemd services related to docker containers aren't active after master node installation:

# systemctl status docker-nailgun
docker-nailgun.service
   Loaded: loaded (/usr/lib/systemd/system/docker-nailgun.service; enabled)
   Active: inactive (dead)

When we run `dockerctl restart nailgun` command in tests it just triggers `systemctl restart docker-nailgun` command and systemd attaches to already running docker container:

Feb 2 22:04:15 nailgun systemd[1]: Starting docker-nailgun.service...
Feb 2 22:04:15 nailgun docker: time="2016-02-02T22:04:15.056670551Z" level=info msg="GET /v1.20/containers/json?all=1"
Feb 2 22:04:15 nailgun systemd[1]: Started docker-nailgun.service.

So Nailgun container isn't restarted. Moving the bug to area-library, because systemd services should be started by puppet during master node deployment. Otherwise services in containers won't be monitored until first master node reboot.