Script for checking containers readiness is buggy

[Mike Scherbakov]

My log:
rsync is ready.
checking container astute
checking with command "shell_container astute ps aux | grep -q 'astuted'"
checking with command "curl -f -L -u naily:naily -s http://127.0.0.1:15672/api/exchanges | grep -qw 'nailgun'"
checking with command "curl -f -L -u naily:naily -s http://127.0.0.1:15672/api/exchanges | grep -qw 'naily'"
try number 1
return code is
try number 2
return code is
try number 3
.....
try number 240
return code is
try number 241
return code is
checking with command "curl -f -L -u naily:naily -s http://127.0.0.1:15672/api/exchanges | grep -qw 'naily_service'"
astute is ready.

Expected:
1) return code is shown, it is not space
2) astute obviously was not ready, as we exceeded all retries.

Issues are in checker code:
https://github.com/stackforge/fuel-library/blob/a69e118146e7d4398a02e3eeff413e008ebb3716/deployment/puppet/docker/templates/functions.sh.erb#L40
for first issue, and for second one - we run all following checks, and from following checks we receive failure=0, which overrides global variable.

[Mike Scherbakov (mihgen)]

Initial issue with checks is being fixed by https://review.openstack.org/#/c/93913/.

[Aleksandr Didenko (adidenko)]

1) is fixed here: https://review.openstack.org/#/c/93914/
2) "we run all following checks, and from following checks we receive failure=0, which overrides global variable". We make decision about filure=1 right after "case" block in function "check_ready" https://github.com/stackforge/fuel-library/blob/a69e118146e7d4398a02e3eeff413e008ebb3716/deployment/puppet/docker/templates/functions.sh.erb#L81 so we do not override it since we do not have any iterations inside "check_ready" funct . Here is a simple test I've just run (dockerctl check):

...
return code is 1
try number 20
return code is 1
try number 21
return code is 1
ERROR: astute failed to start.
checking container rsyslog

[Aleksandr Didenko (adidenko)]

OK, https://review.openstack.org/#/c/93914/2 now also fixes multiple checks for a single container logic:

checking container astute
checking with command "shell_container astute ps aux | grep -q 'astuted'"
checking with command "curl -f -L -u naily:naily -s http://127.0.0.1:15672/api/exchanges | grep -qw 'nailgun'"
checking with command "curl -f -L -u naily:naily -s http://127.0.0.1:15672/api/exchanges | grep -qw 'naily'"
try number 1
return code is
try number 2
return code is
try number 3
return code is
try number 4
return code is
try number 5
return code is
try number 6
return code is
checking with command "curl -f -L -u naily:naily -s http://127.0.0.1:15672/api/exchanges | grep -qw 'naily_service'"
ERROR: astute failed to start.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/93914
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=96bd3f1ce880ad35664729144964dad8cbcc45d2
Submitter: Jenkins
Branch: master

commit 96bd3f1ce880ad35664729144964dad8cbcc45d2
Author: Aleksandr Didenko <email address hidden>
Date: Fri May 16 16:28:13 2014 +0300

    Fix eval exit code in dockectl check function

    Changing "while !" to "until" in order to pass "eval" exit code
    correctly to the output when checking docker containers.

    Also fixing multiple check for a single container logic.

    Also fixing astute container check - "naily" rabbitmq exchange
    appears only after first deployment. It should not be there right
    after Fuel node deployment.

    Closes-bug: #1320241
    Change-Id: I1e5394dae87e490c2b99edb4b069dfedc00b62b6