kolla_keystone_bootstrap occasionally returns no json output, fails the deploy

[Paul Bourke]

Can be observed manually by executing by hand on a control node:

# docker exec -t keystone kolla_keystone_bootstrap admin zgh6CqSoWMxnoacQVkWyRivkVRnyWiDhEO6YlC9j admin admin http://192.168.7.99:35357/v3 http://192.168.7.99:5000/v3 http://192.168.7.99:5000/v3 RegionOne

(relevant args would need to be subbed for your own system)

Strangely running again returns the expected:

{"failed": false, "changed": false}

It seems to fail about every one in two times for me.

Results in the following unintelligible failure to the operator:

TASK: [keystone | Creating admin project, user, role, service, and endpoint] ***
<control01> REMOTE_MODULE command docker exec -t keystone kolla_keystone_bootstrap admin zgh6CqSoWMxnoacQVkWyRivkVRnyWiDhEO6YlC9j admin admin http://192.168.7.99:35357/v3 http://192.168.7.99:5000/v3 http://192.168.7.99:5000/v3 RegionOne
fatal: [control01] => Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/runner/__init__.py", line 589, in _executor
    exec_rc = self._executor_internal(host, new_stdin)
  File "/usr/lib/python2.7/site-packages/ansible/runner/__init__.py", line 793, in _executor_internal
    return self._executor_internal_inner(host, self.module_name, self.module_args, inject, port, complex_args=complex_args)
  File "/usr/lib/python2.7/site-packages/ansible/runner/__init__.py", line 1105, in _executor_internal_inner
    data['changed'] = utils.check_conditional(changed_when, self.basedir, inject, fail_on_undefined=self.error_on_undefined_vars)
  File "/usr/lib/python2.7/site-packages/ansible/utils/__init__.py", line 276, in check_conditional
    conditional = template.template(basedir, conditional, inject, fail_on_undefined=fail_on_undefined)
  File "/usr/lib/python2.7/site-packages/ansible/utils/template.py", line 124, in template
    varname = template_from_string(basedir, varname, templatevars, fail_on_undefined)
  File "/usr/lib/python2.7/site-packages/ansible/utils/template.py", line 382, in template_from_string
    res = jinja2.utils.concat(rf)
  File "<template>", line 9, in root
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

[Paul Bourke (pauldbourke)]

Worth nothing, this may be an oddity in the oraclelinux base, as nobody else as of yet has seen this. Running without the -t seems to fix it but not sure why yet.

[Steven Dake (sdake)]

Paul,

We have seen this numerous times in the gate, but it is very hard to reproduce on my test centos system. that said, I have seen it so its snot just oracle linux.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/307876
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=e2452906f3389e0ae2fa704ee4ce431e1c9f832b
Submitter: Jenkins
Branch: master

commit e2452906f3389e0ae2fa704ee4ce431e1c9f832b
Author: Paul Bourke <email address hidden>
Date: Tue Apr 19 15:59:51 2016 +0100

    Disable tty for keystone register script

    There seems to be a bug regarding the interaction between the Red Hat
    based images and docker exec, where output is missed when attaching a
    tty. This can be replicated using the following:

    $ docker run -d --name test centos /bin/sleep infinity
    $ docker exec -t test echo hi

    Repeat the second command over and over, "hi" will only be printed out
    every couple of runs.

    This affects the keystone register task as sometimes it will not print
    it's final json result to stdout, causing ansible to fall over with a
    difficult to diagnose error (despite the register having run
    successfully).

    Disabling the tty fixes this for me, it should not be needed in this
    case regardless.

    Change-Id: Ie7eb7c01c34ee3c59bd843651195fbcb7259d2c8
    Closes-Bug: #1572082

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309107
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=fc6b00ba6931f658d5e1b6831384d1b69d9b72a7
Submitter: Jenkins
Branch: stable/mitaka

commit fc6b00ba6931f658d5e1b6831384d1b69d9b72a7
Author: Paul Bourke <email address hidden>
Date: Tue Apr 19 15:59:51 2016 +0100

    Disable tty for keystone register script

    There seems to be a bug regarding the interaction between the Red Hat
    based images and docker exec, where output is missed when attaching a
    tty. This can be replicated using the following:

    $ docker run -d --name test centos /bin/sleep infinity
    $ docker exec -t test echo hi

    Repeat the second command over and over, "hi" will only be printed out
    every couple of runs.

    This affects the keystone register task as sometimes it will not print
    it's final json result to stdout, causing ansible to fall over with a
    difficult to diagnose error (despite the register having run
    successfully).

    Disabling the tty fixes this for me, it should not be needed in this
    case regardless.

    Change-Id: Ie7eb7c01c34ee3c59bd843651195fbcb7259d2c8
    Closes-Bug: #1572082
    (cherry picked from commit e2452906f3389e0ae2fa704ee4ce431e1c9f832b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309105
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=ac16c8483ee76ccfa3f1162cc6c30b9249e16c75
Submitter: Jenkins
Branch: stable/liberty

commit ac16c8483ee76ccfa3f1162cc6c30b9249e16c75
Author: Paul Bourke <email address hidden>
Date: Tue Apr 19 15:59:51 2016 +0100

    Disable tty for keystone register script

    There seems to be a bug regarding the interaction between the Red Hat
    based images and docker exec, where output is missed when attaching a
    tty. This can be replicated using the following:

    $ docker run -d --name test centos /bin/sleep infinity
    $ docker exec -t test echo hi

    Repeat the second command over and over, "hi" will only be printed out
    every couple of runs.

    This affects the keystone register task as sometimes it will not print
    it's final json result to stdout, causing ansible to fall over with a
    difficult to diagnose error (despite the register having run
    successfully).

    Disabling the tty fixes this for me, it should not be needed in this
    case regardless.

    Change-Id: Ie7eb7c01c34ee3c59bd843651195fbcb7259d2c8
    Closes-Bug: #1572082
    (cherry picked from commit e2452906f3389e0ae2fa704ee4ce431e1c9f832b)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 2.0.0 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 1.1.0 release.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/kolla 3.0.0.0b1 development milestone.