Mistral thowrs error while running commands

Bug #1808668 reported by Deepanshu Chandna
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mistral
Incomplete
Undecided
Unassigned

Bug Description

Mistral throws error while running commands due to which my actions are always hung.

1. when I execute mistral validate ::

Error on command line -
[user@abc123workflows]$ mistral workflow-validate custom_workflow_common.yaml
ERROR (app) ('Connection aborted.', BadStatusLine("''",))

Error in log file ::

[2018-12-16 00:04:41 +0000] [1919] [ERROR] Socket error processing request.
Traceback (most recent call last):
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 191, in handle_request
    six.reraise(*sys.exc_info())
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
    response = req.get_response(self.application)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 1313, in send
    application, catch_exc_info=False)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 1277, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 840, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 683, in __call__
    self.invoke_controller(controller, args, kwargs, state)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 574, in invoke_controller
    result = controller(*args, **kwargs)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/mistral/api/controllers/v2/validation.py", line 32, in post
    definition = pecan.request.text
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 77, in __getattr__
    return getattr(obj, attr)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 52, in __getattribute__
    return WebObRequest.__getattribute__(self, name)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 731, in _text__get
    body = self.body
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/pecan/core.py", line 52, in __getattribute__
    return WebObRequest.__getattribute__(self, name)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 692, in body
    self.make_body_seekable() # we need this to have content_length
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 928, in make_body_seekable
    self.copy_body()
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 978, in copy_body
    fileobj = self.make_tempfile()
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 1018, in make_tempfile
    return tempfile.TemporaryFile()
  File "/usr/lib64/python2.7/tempfile.py", line 489, in TemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib64/python2.7/tempfile.py", line 239, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/eventlet/green/os.py", line 109, in open
    fd = __original_open__(file, flags, mode)
OSError: [Errno 2] No such file or directory: '/tmp/tmpfdLvge'

================================================

Please suggest .

Revision history for this message
Deepanshu Chandna (deepanshu20) wrote :

Can anyone please suggest here.

Changed in mistral:
status: New → Incomplete
Revision history for this message
Adriano Petrich (apetrich) wrote :

Could you share the contents of the custom_workflow_common.yaml please?

Revision history for this message
Deepanshu Chandna (deepanshu20) wrote :
Download full text (77.6 KiB)

version: '2.0'

hello.custom_workflow_common:
    description: st2 hello.custom_workflow_common
    type: direct
    input:
        - TargetName
        - transaction_name
        - transaction_parameters
        - ObjectServer
        - TOTA
        - Serial
        - timeLimit
        - deployment_execution_id
        - base_path
        - ServerName
        - Zone
    output:
        stdout: <% $.completion_code_stdout %>
    tasks:
        transaction_verify:
            action: core.local
            input:
                cmd: ./<% $.base_path %>/scripts/get_ScriptNameFromTransaction.sh <% $.transaction_name %>
                timeout: <% $.timeLimit %>
            publish:
                transaction_verify_stdout: <% task(transaction_verify).result.stdout %>
            on-success:
                - task_transaction_verify_check
            on-error:
                - fail
        task_transaction_verify_check:
            on-success:
                - initial_state: <% not $.transaction_verify_stdout in ['None','na','',' ']%>
                - transaction_wrong: <% $.transaction_verify_stdout in ['None','na','',' ']%>
        initial_state:
            action: core.local_sudo
            input:
                cmd: <% $.base_path %>/scripts/initialstate_UWF.sh <% $.transaction_name %> <% $.TargetName %> <% $.TOTA %> <% $.Serial %> <% $.deployment_execution_id %> <% $.ServerName %> <% $.Zone %> "2"
                timeout: <% $.timeLimit %>
            publish:
                portalid: <% $.transaction_parameters %>
                status_message: "-----------initial state-------------"
                stderr: <% task(initial_state).result.stderr %>
            on-success:
                - curl_initial_state
            on-error:
                - fail
        transaction_wrong:
            action: core.local_sudo
            input:
                cmd: <% $.base_path %>/scripts/initialstate_UWF.sh <% $.transaction_name %> <% $.TargetName %> <% $.TOTA %> <% $.Serial %> <% $.deployment_execution_id %> <% $.ServerName %> <% $.Zone %> "101"
                timeout: <% $.timeLimit %>
            publish:
                portalid: <% $.transaction_parameters %>
                status_message: "-----------transaction wrong-------------"
                stderr: <% task(transaction_wrong).result.stderr %>
                completion_code_stdout: 101
            on-success:
                - TOTA_STATE_UPDATE
            on-error:
                - fail
        curl_initial_state:
            action: core.local
            input:
                cmd: "<% $.base_path %>/scripts/curlstate_UW.sh <% $.deployment_execution_id %>"
                timeout: <% $.timeLimit %>
            publish:
                stdout_get_initial_state: <% task(curl_initial_state).result.stdout %>
                stderr: <% task(curl_initial_state).result.stderr %>
                status_message: "Script curl_get_notes"
            on-success:
                - curl_initial_state_run
            on-error:
                - fail
        curl_initial_state_run:
            action: core.local
            input:
                cmd: "<% $.stdout_get_initial...

Revision history for this message
Deepanshu Chandna (deepanshu20) wrote :

Also, mistral-server is eating up all the CPU. Please help - How to kill all the actions running in mistral workflow or any other solution.
 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 14304 mistral 20 0 400836 95684 6968 R 98.3 0.3 0:20.71 mistral-server
 14206 st2 20 0 327088 54864 6428 S 2.3 0.2 0:01.73 st2notifier

Revision history for this message
Deepanshu Chandna (deepanshu20) wrote :

$ tail -f mistral-api.log
    fileobj = self.make_tempfile()
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/webob/request.py", line 1018, in make_tempfile
    return tempfile.TemporaryFile()
  File "/usr/lib64/python2.7/tempfile.py", line 489, in TemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags)
  File "/usr/lib64/python2.7/tempfile.py", line 239, in _mkstemp_inner
    fd = _os.open(file, flags, 0600)
  File "/opt/stackstorm/mistral/lib/python2.7/site-packages/eventlet/green/os.py", line 109, in open
    fd = __original_open__(file, flags, mode)
OSError: [Errno 2] No such file or directory: '/tmp/tmpGoLWVx'

Can you help me resolving this error - Please.

Revision history for this message
Deepanshu Chandna (deepanshu20) wrote :
Download full text (4.0 KiB)

I also observed - I do not get above error on my other server which is RHEL 6 . I checked "ps -ef | grep mistral"found a small difference - Server which is failing does not have "/var/run/mistral/mistral-api.pid"

Server where mistral runs fine -

```
ps -ef | grep mistral
postgres 7240 7090 9 21:43 ? 00:05:25 postgres: mistral mistral 127.0.0.1(53346) idle
mistral 7445 1 0 2018 ? 00:15:20 /opt/stackstorm/mistral/bin/python /opt/stackstorm/mistral/bin/gunicorn --log-file /var/log/mistral/mistral-api.log -b 127.0.0.1:8989 -w 2 mistral.api.wsgi --graceful-timeout 10 --pid /var/run/mistral/mistral-api.pid
mistral 7506 1 18 2018 ? 10-17:17:49 /opt/stackstorm/mistral/bin/python /opt/stackstorm/mistral/bin/mistral-server --server engine,executor,notifier --config-file /etc/mistral/mistral.conf --log-file /var/log/mistral/mistral-server.log
postgres 7535 7090 0 21:45 ? 00:00:02 postgres: mistral mistral 127.0.0.1(53508) idle
postgres 8335 7090 0 21:51 ? 00:00:01 postgres: mistral mistral 127.0.0.1(54070) idle
postgres 8525 7090 0 21:53 ? 00:00:01 postgres: mistral mistral 127.0.0.1(54194) idle
mistral 8876 7445 2 2018 ? 1-14:17:09 /opt/stackstorm/mistral/bin/python /opt/stackstorm/mistral/bin/gunicorn --log-file /var/log/mistral/mistral-api.log -b 127.0.0.1:8989 -w 2 mistral.api.wsgi --graceful-timeout 10 --pid /var/run/mistral/mistral-api.pid
mistral 8877 7445 2 2018 ? 1-14:46:51 /opt/stackstorm/mistral/bin/python /opt/stackstorm/mistral/bin/gunicorn --log-file /var/log/mistral/mistral-api.log -b 127.0.0.1:8989 -w 2 mistral.api.wsgi --graceful-timeout 10 --pid /var/run/mistral/mistral-api.pid
postgres 16168 7090 9 22:09 ? 00:02:56 postgres: mistral mistral 127.0.0.1(55562) idle
postgres 16413 7090 9 22:11 ? 00:02:44 postgres: mistral mistral 127.0.0.1(55784) idle
postgres 16429 7090 9 22:11 ? 00:02:42 postgres: mistral mistral 127.0.0.1(55786) idle
postgres 17522 7090 9 22:21 ? 00:01:52 postgres: mistral mistral 127.0.0.1(56528) idle
postgres 17528 7090 9 22:21 ? 00:01:50 postgres: mistral mistral 127.0.0.1(56534) idle
postgres 17801 7090 9 22:23 ? 00:01:38 postgres: mistral mistral 127.0.0.1(56748) idle
postgres 21356 7090 0 22:39 ? 00:00:00 postgres: mistral mistral 127.0.0.1(58096) idle
```

servers where problem occurs -

```
 ps -ef | grep mistral
mistral 5076 1 0 2018 ? 00:02:29 /opt/stackstorm/mistral/bin/python /opt/stackstorm/mistral/bin/gunicorn --log-file /var/log/mistral/mistral-api.log -b 127.0.0.1:8989 -w 2 mistral.api.wsgi --graceful-timeout 10
mistral 5077 1 9 2018 ? 1-19:00:36 /opt/stackstorm/mistral/bin/python /opt/stackstorm/mistral/bin/mistral-server --server engine,executor,notifier --config-file /etc/mistral/mistral.conf --log-file /var/log/mistral/mistral-server.log
mistral 5109 5076 1 2018 ? 05:38:08 /opt/stackstorm/mistral/bin/python /opt/stackstorm/mistral/bin/gunicorn --log-file /var/log/mistral/mistral-api.log -b 127.0.0.1:8989 -w 2 mistral.api.wsgi --graceful-timeout 10
mistral 5110 5076 1 2018 ? 05:34:...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.