| 2016-03-01 19:54:22 |
Caio Begotti |
description |
Only after turning Mojo's debug on I figured out why its verify phase was failing catastrophically: my service went rogue for a few minutes generating over 4k oops, which is way above the $(ulimit -n) of the unit, so when Mojo ran the verify phase it failed due to other issues but then it errored out when pulling the logs. I suppose limiting how many logs Mojo would pull is not a solution -- well, unless there is an easy parameter for us to use in debug-logs -- so maybe at least it should not error out like that based on the current $(ulimit -n)?
$ mojo verify --options config=verify-nagios
2016-02-29 21:14:57 [INFO] Using pre-existing container class lxc
2016-02-29 21:14:57 [INFO] Pulling secrets from /srv/mojo/LOCAL/.../staging to /srv/mojo/.../staging/local
2016-02-29 21:14:57 [INFO] Running script verify-nagios
2016-02-29 21:21:46 [DEBUG] Debug Log Config found: {'service-cache': {'logs': ['/srv/service/*/logs/www-oops/*/OOPS-*']}}
2016-02-29 21:22:17 [ERROR] Unknown error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/mojo/cli.py", line 503, in run_with_args
args.func(args)
File "/usr/lib/python2.7/dist-packages/mojo/utils.py", line 305, in wrapped
return method(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/mojo/cli.py", line 32, in run_phase
phase.run(project, workspace, args.stage)
File "/usr/lib/python2.7/dist-packages/mojo/phase.py", line 716, in run
exception_output = "{}\n{}".format(e.output, dl.get_log_output())
File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 64, in get_log_output
for log in self._gather_logs():
File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 116, in _gather_logs
content = self._run_remote_command(unit, command, sanitized_path_list)
File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 101, in _run_remote_command
return subprocess.check_output(cmd, stderr=subprocess.STDOUT)
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 7] Argument list too long |
Only after turning Mojo's debug on I figured out why its verify phase was failing catastrophically: my service went rogue for a few minutes generating over 4k oops, which is way above the $(ulimit -n) of the unit, so when Mojo ran the verify phase it failed due to other issues but then it errored out when pulling the logs. I suppose limiting how many logs Mojo would pull is not a solution -- well, unless there is an easy parameter for us to use in debug-logs -- so maybe at least it should not error out like that based on the current $(ulimit -n)?
$ mojo --log_level DEBUG verify --options config=verify-nagios
2016-02-29 21:14:57 [INFO] Using pre-existing container class lxc
2016-02-29 21:14:57 [INFO] Pulling secrets from /srv/mojo/LOCAL/.../staging to /srv/mojo/.../staging/local
2016-02-29 21:14:57 [INFO] Running script verify-nagios
2016-02-29 21:21:46 [DEBUG] Debug Log Config found: {'service-cache': {'logs': ['/srv/service/*/logs/www-oops/*/OOPS-*']}}
2016-02-29 21:22:17 [ERROR] Unknown error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/mojo/cli.py", line 503, in run_with_args
args.func(args)
File "/usr/lib/python2.7/dist-packages/mojo/utils.py", line 305, in wrapped
return method(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/mojo/cli.py", line 32, in run_phase
phase.run(project, workspace, args.stage)
File "/usr/lib/python2.7/dist-packages/mojo/phase.py", line 716, in run
exception_output = "{}\n{}".format(e.output, dl.get_log_output())
File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 64, in get_log_output
for log in self._gather_logs():
File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 116, in _gather_logs
content = self._run_remote_command(unit, command, sanitized_path_list)
File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 101, in _run_remote_command
return subprocess.check_output(cmd, stderr=subprocess.STDOUT)
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 7] Argument list too long |
|
