Mistral can be too limiting regarding action output length

Bug #1718384 reported by Jiří Stránský
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Dougal Matthews

Bug Description

When running Ansible via a Mistral action, it's fairly easy to hit the size limits in Mistral for stdout length. The error message looks as such:

2017-09-19 12:31:40.140 21170 INFO workflow_trace [-] Task 'deploy_kubernetes' (7433eeee-60b3-4cce-b81a-cbf0d7787bad) [RUNNING -> ERROR, msg=Failed to complete action [error=Size of 'output' is 4907KB which exceeds the limit of 4096KB, action=tripleo.ansible-playbook, task=deploy_kubernetes]:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/mistral/engine/action_handler.py", line 37, in on_action_complete
    action.complete(result)
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/mistral/engine/actions.py", line 227, in complete
    self.action_ex.output = self._prepare_output(result).to_dict()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 224, in __set__
    instance_dict(instance), value, None)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 702, in set
    value, old, initiator)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 711, in fire_replace_event
    self._init_append_or_replace_token())
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/events.py", line 1884, in wrap
    fn(target, value, *arg)
  File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 92, in <lambda>
    lambda t, v, o, i: validate_long_type_length(cls, attr_name, v)
  File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 77, in validate_long_type_length
    size_limit_kb
SizeLimitExceededException: Size of 'output' is 4907KB which exceeds the limit of 4096KB
] (execution_id=90f62e35-0acb-4839-a086-5d57de830876)

So far the workaround has been to run ansible with `verbosity: 0`.

Regardless of this particular use case, it may be worth evaluating if Mistral action output limits shouldn't be higher.

Dougal Matthews (d0ugal)
no longer affects: mistral
Changed in tripleo:
assignee: nobody → Dougal Matthews (d0ugal)
status: Triaged → In Progress
Revision history for this message
Dougal Matthews (d0ugal) wrote :

I removed Mistral as it is already configurable, so just a tripleo issue.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to instack-undercloud (master)

Fix proposed to branch: master
Review: https://review.openstack.org/505582

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to instack-undercloud (master)

Reviewed: https://review.openstack.org/505582
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=83bf89ee590e13afc88281fc0ef52e45a77f7cb4
Submitter: Jenkins
Branch: master

commit 83bf89ee590e13afc88281fc0ef52e45a77f7cb4
Author: Dougal Matthews <email address hidden>
Date: Wed Sep 20 11:03:54 2017 +0100

    Increase the size of the Mistral output limit

    This increases it from 4 MB to 16.

    Closes-Bug: #1718384
    Change-Id: Ief9e8402f0251d1447a825f496050334a8d80dc5

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to instack-undercloud (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/505819

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to instack-undercloud (stable/pike)

Reviewed: https://review.openstack.org/505819
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=c97752b9944e53dd439724412d5dd8d5afa45564
Submitter: Jenkins
Branch: stable/pike

commit c97752b9944e53dd439724412d5dd8d5afa45564
Author: Dougal Matthews <email address hidden>
Date: Wed Sep 20 11:03:54 2017 +0100

    Increase the size of the Mistral output limit

    This increases it from 4 MB to 16.

    Closes-Bug: #1718384
    Change-Id: Ief9e8402f0251d1447a825f496050334a8d80dc5
    (cherry picked from commit 83bf89ee590e13afc88281fc0ef52e45a77f7cb4)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/instack-undercloud 7.4.1

This issue was fixed in the openstack/instack-undercloud 7.4.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/instack-undercloud 8.0.0

This issue was fixed in the openstack/instack-undercloud 8.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/516771

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/516771
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=71dad1a001e4c4af9a5765e3fd9d63993bcfe6a6
Submitter: Zuul
Branch: master

commit 71dad1a001e4c4af9a5765e3fd9d63993bcfe6a6
Author: Dan Prince <email address hidden>
Date: Tue Oct 31 15:28:22 2017 -0400

    Increase the size of the Mistral output limit

    This patch adds a new MistralExecutionFieldSizeLimit which
    can be used to control the Mistral output limit.

    Also, increases the default on the undercloud to 16MB.

    Change-Id: If101566a22c7cce4af92c890cb9eb29e146b1599
    Related-bug: #1718384

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.