Comment 0 for bug 1551931

Revision history for this message
Caio Begotti (caio1982) wrote :

Only after turning Mojo's debug on I figured out why its verify phase was failing catastrophically: my service went rogue for a few minutes generating over 4k oops, which is way above the $(ulimit -n) of the unit, so when Mojo ran the verify phase it failed due to other issues but then it errored out when pulling the logs. I suppose limiting how many logs Mojo would pull is not a solution -- well, unless there is an easy parameter for us to use in debug-logs -- so maybe at least it should not error out like that based on the current $(ulimit -n)?

$ mojo verify --options config=verify-nagios
2016-02-29 21:14:57 [INFO] Using pre-existing container class lxc
2016-02-29 21:14:57 [INFO] Pulling secrets from /srv/mojo/LOCAL/.../staging to /srv/mojo/.../staging/local
2016-02-29 21:14:57 [INFO] Running script verify-nagios
2016-02-29 21:21:46 [DEBUG] Debug Log Config found: {'service-cache': {'logs': ['/srv/service/*/logs/www-oops/*/OOPS-*']}}
2016-02-29 21:22:17 [ERROR] Unknown error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/mojo/cli.py", line 503, in run_with_args
    args.func(args)
  File "/usr/lib/python2.7/dist-packages/mojo/utils.py", line 305, in wrapped
    return method(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mojo/cli.py", line 32, in run_phase
    phase.run(project, workspace, args.stage)
  File "/usr/lib/python2.7/dist-packages/mojo/phase.py", line 716, in run
    exception_output = "{}\n{}".format(e.output, dl.get_log_output())
  File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 64, in get_log_output
    for log in self._gather_logs():
  File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 116, in _gather_logs
    content = self._run_remote_command(unit, command, sanitized_path_list)
  File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 101, in _run_remote_command
    return subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 7] Argument list too long