Long output from ceph-ansible seems to be able to break CLI

[Jiří Stránský]

The root cause is still not 100% confirmed so take the description with a grain of salt: We hit an issue where ceph-ansible runs successfully, and the wrapping mistral-workflow runs successfully too, but the CLI command crashes as soon as it receives ceph-ansible output. I think this might be caused by "run ceph-ansible" task sending a giant single-line output into the parent Ansible log.

Additional info at https://bugzilla.redhat.com/show_bug.cgi?id=1631382

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/607302

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/607588

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/607302
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a0ead2f1a794a14cdde67b652decbb1376f3a174
Submitter: Zuul
Branch: master

commit a0ead2f1a794a14cdde67b652decbb1376f3a174
Author: Jiri Stransky <email address hidden>
Date: Tue Oct 2 18:14:47 2018 +0200

    Chunk up ceph-ansible output to prevent clogging the CLI

    The CLI commands running Ansible can crash if we send too much
    single-line log output their way. This was happening on upgrades, when
    we run Ansible with verbosity level 1.

    The fix is twofold:

    * If ceph-ansible finishes successfully, we don't print the
      ceph-ansible output into the main log.

    * If ceph-ansible fails, we do print the output, but we print it
      line-by-line, which should give us much better readability than
      before, and we shouldn't break the limits of the Mistral-Zaqar-CLI
      message passing.

    Change-Id: I6e0fc36749e74fce25f414c2547e49e2a20437ab
    Closes-Bug: #1795689

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/607588
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=aed9d771a2b080021ef61de9120a4bdf48dcc63e
Submitter: Zuul
Branch: stable/rocky

commit aed9d771a2b080021ef61de9120a4bdf48dcc63e
Author: Jiri Stransky <email address hidden>
Date: Tue Oct 2 18:14:47 2018 +0200

    Chunk up ceph-ansible output to prevent clogging the CLI

    The CLI commands running Ansible can crash if we send too much
    single-line log output their way. This was happening on upgrades, when
    we run Ansible with verbosity level 1.

    The fix is twofold:

    * If ceph-ansible finishes successfully, we don't print the
      ceph-ansible output into the main log.

    * If ceph-ansible fails, we do print the output, but we print it
      line-by-line, which should give us much better readability than
      before, and we shouldn't break the limits of the Mistral-Zaqar-CLI
      message passing.

    Change-Id: I6e0fc36749e74fce25f414c2547e49e2a20437ab
    Closes-Bug: #1795689
    (cherry picked from commit a0ead2f1a794a14cdde67b652decbb1376f3a174)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.1.0 release.