Enabling too many environments from the UI breaks Mistral

[Julie Pichon]

I'm using the UI to configure a deployment.

Steps to reproduce:
-------------------
1. Select a bunch of environments. I tried with 4 in this case: network isolation v6, storage environment, storage environment in order to deploy Ceph, enable tls, tls endpoints public IPs. I have 1 Controller, 1 Compute, 1 Ceph Node assigned.

Problem description:
--------------------
After saving the new environments, it becomes impossible to use the UI to configure any parameters:

- If I click on Parameters in "Edit Configuration", I get the message "There are currently no parameters to configure in this section"

From the browser console:

action @ 13:48:38.418 FETCH_PARAMETERS_PENDING
reqwest.js:240 POST http://192.0.2.1:8989/v2/action_executions 500 (Internal Server Error)
logger.js:66 action @ 13:48:49.833 FETCH_PARAMETERS_FAILED
logger.js:58 Error in ParametersActions.fetchParameters
XMLHttpRequest {readyState: 4, timeout: 0, withCredentials: false, upload: XMLHttpRequestUpload, responseURL: "http://192.0.2.1:8989/v2/action_executions"…}

- If I click on any edit "Pencil" button for any card, all 3 tabs - Parameters, Services, Network Configuration - shows the same "no parameters" error message.

- If I click on "Edit Configuration" and wait for the "Overall settings to load", the window closes on its own with the message "{"debuginfo": null, "faultcode": "Server", "faultstring": "__init__() takes exactly 4 arguments (2 given)"}"

Mistral logs
------------

This error appears to be coming from Mistral directly.

Looking at the api.log:

2016-11-10 13:51:57.757 11167 INFO mistral.api.controllers.v2.action_execution [-] Create action_execution [action_execution=ActionExecution [id='Unset', workflow_name='Unset', task_name='Unset', task_execution_id='Unset', state='Unset', state_info='Unset', tags='Unset', name='tripleo.heat_capabilities.get', description='Unset', accepted='Unset', input='{'container': 'test'}', output='Unset', created_at='Unset', updated_at='Unset', params='{'save_result': True, 'run_sync': True}']]
2016-11-10 13:52:03.057 11167 INFO mistral.api.controllers.v2.environment [-] Fetch environment [name=test]
2016-11-10 13:52:07.666 11167 ERROR wsme.api [-] Server-side error: "__init__() takes exactly 4 arguments (2 given)". Detail:
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/wsmeext/pecan.py", line 84, in callfunction
    result = f(self, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/mistral/api/controllers/v2/action_execution.py", line 141, in post
    **params

  File "/usr/lib/python2.7/site-packages/mistral/engine/rpc_backend/rpc.py", line 317, in decorator
    raise exc_cls(e.value)

TypeError: __init__() takes exactly 4 arguments (2 given)

That one looks like a bug on the Mistral side?

The engine logs show the following:

2016-11-10 13:52:07.657 11554 ERROR mistral.db.v2.sqlalchemy.models [-] Size limit 1024KB exceed for class [<class 'mistral.db.v2.sqlalchemy.models.TaskExecution'>], field output of size 1148KB.
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server [-] Exception during message handling
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 225, in dispatch
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 195, in _do_dispatch
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/mistral/engine/rpc_backend/rpc.py", line 177, in start_action
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server **params
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/mistral/engine/action_queue.py", line 70, in decorate
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server res = func(*args, **kw)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/mistral/utils/__init__.py", line 123, in _logged
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server return func(*args, **kw)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/mistral/engine/default_engine.py", line 110, in start_action
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server return db_api.create_action_execution(values)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/mistral/db/v2/api.py", line 224, in create_action_execution
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server return IMPL.create_action_execution(values)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/mistral/db/sqlalchemy/base.py", line 118, in _within_session
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server result = func(*args, **kw)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/api.py", line 662, in create_action_execution
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server a_ex.update(values.copy())
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/models.py", line 94, in update
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server setattr(self, k, v)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 224, in __set__
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server instance_dict(instance), value, None)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 701, in set
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server value, old, initiator)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 710, in fire_replace_event
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server self._init_append_or_replace_token())
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/events.py", line 1606, in wrap
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server fn(target, value, *arg)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 92, in <lambda>
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server lambda t, v, o, i: validate_long_type_length(cls, attr_name, v)
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/mistral/db/v2/sqlalchemy/models.py", line 77, in validate_long_type_length
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server size_limit_kb
2016-11-10 13:52:07.659 11554 ERROR oslo_messaging.rpc.server SizeLimitExceededException: Size of 'output' is 1148KB which exceeds the limit of 1024KB

It looks like we may need to adjust the Mistral configuration on the undercloud.

Additional information:
------------------------
$ mistral environment-get test
+-------------+-----------------------------------------------------------------------------------+
| Field | Value |
+-------------+-----------------------------------------------------------------------------------+
| Name | test |
| Description | <none> |
| Variables | { |
| | "passwords": { |
(snip)
    |
| | "parameter_defaults": { |
| | "CloudDomain": "", |
| | "CephStorageCount": "1" |
| | }, |
| | "template": "overcloud.yaml", |
| | "environments": [ |
| | { |
| | "path": "overcloud-resource-registry-puppet.yaml" |
| | }, |
| | { |
| | "path": "environments/network-isolation-v6.yaml" |
| | }, |
| | { |
| | "path": "environments/storage-environment.yaml" |
| | }, |
| | { |
| | "path": "environments/enable-tls.yaml" |
| | }, |
| | { |
| | "path": "environments/tls-endpoints-public-ip.yaml" |
| | } |
| | ] |
| | } |
| Scope | private |
| Created at | 2016-11-02 11:39:53 |
| Updated at | 2016-11-10 13:43:13 |

This reproduced on both a Newton and Ocata environment.

[Florian Fuchs (flo-fuchs)]

I hit that error when one of the TLS endpoint environments (dns name or ip address) was selected. When I didn't use these two (by unchecking in "Overall settings" in the UI), the parameters could be fetched without problems.

[Julie Pichon (jpichon)]

I became able to access the parameters again after increasing the Mistral execution size.

Workaround
-----------
1. Edit /etc/mistral/mistral.conf
[engine]
execution_field_size_limit_kb=4096 (<- any number bigger than the 1024 default)
2. Restart Mistral services
$ sudo systemctl restart openstack-mistral-*

I actually had a patch to increase that limit for other reasons at https://review.openstack.org/#/c/348216/ , I'll restore and update it.

[Julie Pichon (jpichon)]

Attempting running a similar deployment on the command line:

$ openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/enable-tls.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/tls-endpoints-public-ip.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation-v6.yaml

And the environment then looks like this:

| | "environments": [ |
| | { |
| | "path": "overcloud-resource-registry-puppet.yaml" |
| | }, |
| | { |
| | "path": "user-environment.yaml" |
| | }

So it looks like the CLI is bunching all the environments together into one big file, and that would be why the issue isn't coming up in that case.

[Julie Pichon (jpichon)]

Increasing the Importance as it's impossible to do even slightly complex deployments from the UI without this.

[Julie Pichon (jpichon)]

Patch at https://review.openstack.org/#/c/348216/

[Julie Pichon (jpichon)]

I filed the Mistral API 500 error at bug 1640852, for reference.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/348216
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=4ee68a5229bed653a669826e45108d25f207c024
Submitter: Jenkins
Branch: master

commit 4ee68a5229bed653a669826e45108d25f207c024
Author: Julie Pichon <email address hidden>
Date: Thu Jul 28 10:56:01 2016 +0100

    Increase Mistral Task Size limit

    When selecting environments to deploy from the UI, every environment
    filename gets added individually to 'environments' key in the Mistral
    environment. Apparently it means we hit the Mistral execution size
    limit pretty quickly.

    This isn't needed for the CLI at this point because it bunches all the
    environment files together into one big "user-environments.yaml" file,
    for backwards-compabitility reasons.

    Note that this Mistral value seems to affect both input and output for
    task execution, which is why this patch was initially created as a
    prerequisite for Ia9d2312e942229b659cfc65d4c370874e87f44da.

    Change-Id: I24453b6d83171860065b050f437602e1f655d8b8
    Closes-Bug: #1640815

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/396523

[Julie Pichon (jpichon)]

For general reference, summarised from IRC: the CLI eventually will store the environments in the same way than the UI (aka will need this fix too), but cannot at the moment for backwards-compatibility reasons. Custom environments deployed with the CLI can include relative paths [1], which means an env could reference files anywhere locally. For this to continue to work, heatclient is used to discovered all the referenced files, which is how we end up with a merged environment.

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/environments/net-bond-with-vlans.yaml

[Julie Pichon (jpichon)]

The CLI issue mentioned in comment #9 is filed at bug 1640861.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/396523
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=9b22c1072c292a8777a8ab9590262c4332c1c481
Submitter: Jenkins
Branch: stable/newton

commit 9b22c1072c292a8777a8ab9590262c4332c1c481
Author: Julie Pichon <email address hidden>
Date: Thu Jul 28 10:56:01 2016 +0100

    Increase Mistral Task Size limit

    When selecting environments to deploy from the UI, every environment
    filename gets added individually to 'environments' key in the Mistral
    environment. Apparently it means we hit the Mistral execution size
    limit pretty quickly.

    This isn't needed for the CLI at this point because it bunches all the
    environment files together into one big "user-environments.yaml" file,
    for backwards-compabitility reasons.

    Note that this Mistral value seems to affect both input and output for
    task execution, which is why this patch was initially created as a
    prerequisite for Ia9d2312e942229b659cfc65d4c370874e87f44da.

    Change-Id: I24453b6d83171860065b050f437602e1f655d8b8
    Closes-Bug: #1640815
    (cherry picked from commit 4ee68a5229bed653a669826e45108d25f207c024)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack-undercloud 6.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/instack-undercloud 5.2.0 release.