Start dockerctl multiple commands hangs up all of them

[Denis Klepikov]

We need to be sure that container starts and ready to use.
It can be applied to test environment on virualbox and weak machines.

How to reproduce:

open 2 fuel terminals
In first terminal stop nailgun

#dockerctl stop nailgun

Shure that it was stopped

# dockerctl list

Start check of nailgun container

# dockerctl check nailgun
checking container nailgun
checking with command "shell_container nailgun supervisorctl status nailgun | grep -q RUNNING"
lxc-attach: failed to get the init pid
try number 1
return code is 1
lxc-attach: failed to get the init pid
try number 2
return code is 1
lxc-attach: failed to get the init pid
try number 3
return code is 1
........

In another terminal start container with command

# dockerctl start nailgun

we will see next:

fuel-core-5.1-nailgun
checking container nailgun
checking with command "shell_container nailgun supervisorctl status nailgun | grep -q RUNNING"
try number 1
return code is 1
try number 2
return code is 1
try number 3
return code is 1
try number 4

in the first terminal next:

lxc-attach: failed to get the init pid
try number 55
return code is 1
lxc-attach: failed to get the init pid
try number 56
return code is 1
lxc-attach: failed to get the init pid
try number 57

It will continue up to stop by Ctrl+C or command in first terminal

How to reproduce #2:

cold start fuel master
ssh up
login in
# dockerctl check nailgun
checking container nailgun
checking with command "shell_container nailgun supervisorctl status nailgun | grep -q RUNNING"
lxc-attach: failed to get the init pid
try number 1
return code is 1
lxc-attach: failed to get the init pid
try number 2
return code is 1
lxc-attach: failed to get the init pid
try number 3
return code is 1
lxc-attach: failed to get the init pid
try number 4
return code is 1
lxc-attach: failed to get the init pid
try number 5
return code is 1
lxc-attach: failed to get the init pid
try number 6
return code is 1
lxc-attach: failed to get the init pid

try number 7
return code is 1
lxc-attach: failed to get the init pid
try number 8
return code is 1
lxc-attach: failed to get the init pid

next terminal
# ps ax | grep docker | grep nail
 1236 ? S 0:00 /bin/bash /usr/bin/dockerctl start nailgun --attach
 6038 pts/18 S+ 0:00 /bin/bash /usr/bin/dockerctl check nailgun

hue that it was stopped

# dockerctl list

Start check of nailgun container

# dockerctl check nailgun
It will continue due to we stop by Ctrl+C our command in first terminal
and look

# ps ax | grep docker | grep nail
18711 ? S 0:00 /bin/bash /usr/bin/dockerctl start nailgun --attach
18866 ? Sl 0:00 docker -D attach fuel-core-5.1-nailgun

in few seconds we will see
18866 ? Sl 0:00 docker -D attach fuel-core-5.1-nailgun

any another copy of dockerctl blocks first one

Temporary solution:

Copy it to dockerctl-test

cp /usr/bin/dockerctl /usr/bin/dockerctl-test

than we have good testing area

terminal #1

#dockerctl stop nailgun

Shure that it was stopped

# dockerctl list

Start check of nailgun container

# dockerctl check nailgun

#dockerctl start nailgun

In terminal #2
# dockerctl-test check nailgun
checking container nailgun
checking with command "shell_container nailgun supervisorctl status nailgun | grep -q RUNNING"
lxc-attach: failed to get the init pid
try number 1
return code is 1
lxc-attach: failed to get the init pid
try number 2
return code is 1
lxc-attach: failed to get the init pid
try number 3
return code is 1
lxc-attach: failed to get the init pid
try number 4
return code is 1
lxc-attach: failed to get the init pid
try number 5
return code is 1
lxc-attach: failed to get the init pid
try number 6
return code is 1
lxc-attach: failed to get the init pid
try number 7
return code is 1
lxc-attach: failed to get the init pid
try number 8
return code is 1
lxc-attach: failed to get the init pid
try number 9
return code is 1
lxc-attach: failed to get the init pid
try number 10
return code is 1
try number 11
return code is 1
try number 12
return code is 1
try number 13
return code is 1
try number 14
return code is 1
try number 15
return code is 1
try number 16
return code is 1
try number 17
return code is 1
try number 18
return code is 1
try number 19
return code is 1
try number 20
return code is 1
try number 21
return code is 1
try number 22
return code is 1
try number 23
return code is 1
try number 24
return code is 1
try number 25
return code is 1
try number 26
return code is 1
try number 27
return code is 1
try number 28
return code is 1
try number 29
return code is 1
try number 30
return code is 1
try number 31
return code is 1
try number 32
return code is 1
nailgun is ready.

and we can be sure that nailgun container starts and ready to use.

[Dmitry Borodaenko (angdraug)]

Please do not target Medium and lower priority bugs to maintenance releases.

[Sergii Golovatiuk (sgolovatiuk)]

Locking mechanism in dockerctl should be introduce to resolve this bug. It requires refactoring. Moving this bug to 8.0

[Matthew Mosesohn (raytrac3r)]

This is a feature request to enable locks in dockerctl

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/234192

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/234192
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=fd100124b8b2fcfd2c534e11137fcb35bd7070ca
Submitter: Jenkins
Branch: master

commit fd100124b8b2fcfd2c534e11137fcb35bd7070ca
Author: Bartłomiej Piotrowski <email address hidden>
Date: Tue Oct 13 14:54:59 2015 +0200

    Prevent parallel execution of some dockerctl subcommands

    The commit causes dockerctl to fail if another instance of script
    is already running a container-changing function, i.e. export, start,
    destroy, backup and restore. A global variable is used to allow
    high-level functions like backup correctly re-use the lock for start
    or destroy functions.

    Change-Id: I5dc99adeacb901f11ea576674b4fdc64868b8784
    Closes-bug: 1386578

[Artem Hrechanychenko (agrechanichenko)]

Verified on #61 ISO
Checked both variants of reproducing, on virtualbox and kvm.
All works fine.