Debug-logs exception when too many logs are passed on to subprocess

Bug #1551931 reported by Caio Begotti
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mojo: Continuous Delivery for Juju
Confirmed
Medium
Unassigned

Bug Description

Only after turning Mojo's debug on I figured out why its verify phase was failing catastrophically: my service went rogue for a few minutes generating over 4k oops, which is way above the $(ulimit -n) of the unit, so when Mojo ran the verify phase it failed due to other issues but then it errored out when pulling the logs. I suppose limiting how many logs Mojo would pull is not a solution -- well, unless there is an easy parameter for us to use in debug-logs -- so maybe at least it should not error out like that based on the current $(ulimit -n)?

$ mojo --log_level DEBUG verify --options config=verify-nagios
2016-02-29 21:14:57 [INFO] Using pre-existing container class lxc
2016-02-29 21:14:57 [INFO] Pulling secrets from /srv/mojo/LOCAL/.../staging to /srv/mojo/.../staging/local
2016-02-29 21:14:57 [INFO] Running script verify-nagios
2016-02-29 21:21:46 [DEBUG] Debug Log Config found: {'service-cache': {'logs': ['/srv/service/*/logs/www-oops/*/OOPS-*']}}
2016-02-29 21:22:17 [ERROR] Unknown error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/mojo/cli.py", line 503, in run_with_args
    args.func(args)
  File "/usr/lib/python2.7/dist-packages/mojo/utils.py", line 305, in wrapped
    return method(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/mojo/cli.py", line 32, in run_phase
    phase.run(project, workspace, args.stage)
  File "/usr/lib/python2.7/dist-packages/mojo/phase.py", line 716, in run
    exception_output = "{}\n{}".format(e.output, dl.get_log_output())
  File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 64, in get_log_output
    for log in self._gather_logs():
  File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 116, in _gather_logs
    content = self._run_remote_command(unit, command, sanitized_path_list)
  File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 101, in _run_remote_command
    return subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 7] Argument list too long

Caio Begotti (caio1982)
description: updated
Tom Haddon (mthaddon)
Changed in mojo:
status: New → Confirmed
importance: Undecided → Medium
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.