Only after turning Mojo's debug on I figured out why its verify phase was failing catastrophically: my service went rogue for a few minutes generating over 4k oops, which is way above the $(ulimit -n) of the unit, so when Mojo ran the verify phase it failed due to other issues but then it errored out when pulling the logs. I suppose limiting how many logs Mojo would pull is not a solution -- well, unless there is an easy parameter for us to use in debug-logs -- so maybe at least it should not error out like that based on the current $(ulimit -n)?
$ mojo verify --options config=verify-nagios
2016-02-29 21:14:57 [INFO] Using pre-existing container class lxc
2016-02-29 21:14:57 [INFO] Pulling secrets from /srv/mojo/LOCAL/.../staging to /srv/mojo/.../staging/local
2016-02-29 21:14:57 [INFO] Running script verify-nagios
2016-02-29 21:21:46 [DEBUG] Debug Log Config found: {'service-cache': {'logs': ['/srv/service/*/logs/www-oops/*/OOPS-*']}}
2016-02-29 21:22:17 [ERROR] Unknown error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/mojo/cli.py", line 503, in run_with_args
args.func(args)
File "/usr/lib/python2.7/dist-packages/mojo/utils.py", line 305, in wrapped
return method(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/mojo/cli.py", line 32, in run_phase
phase.run(project, workspace, args.stage)
File "/usr/lib/python2.7/dist-packages/mojo/phase.py", line 716, in run
exception_output = "{}\n{}".format(e.output, dl.get_log_output())
File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 64, in get_log_output
for log in self._gather_logs():
File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 116, in _gather_logs
content = self._run_remote_command(unit, command, sanitized_path_list)
File "/usr/lib/python2.7/dist-packages/mojo/juju/debuglogs.py", line 101, in _run_remote_command
return subprocess.check_output(cmd, stderr=subprocess.STDOUT)
File "/usr/lib/python2.7/subprocess.py", line 566, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 7] Argument list too long
Only after turning Mojo's debug on I figured out why its verify phase was failing catastrophically: my service went rogue for a few minutes generating over 4k oops, which is way above the $(ulimit -n) of the unit, so when Mojo ran the verify phase it failed due to other issues but then it errored out when pulling the logs. I suppose limiting how many logs Mojo would pull is not a solution -- well, unless there is an easy parameter for us to use in debug-logs -- so maybe at least it should not error out like that based on the current $(ulimit -n)?
$ mojo verify --options config= verify- nagios LOCAL/. ../staging to /srv/mojo/ .../staging/ local service/ */logs/ www-oops/ */OOPS- *']}} python2. 7/dist- packages/ mojo/cli. py", line 503, in run_with_args python2. 7/dist- packages/ mojo/utils. py", line 305, in wrapped python2. 7/dist- packages/ mojo/cli. py", line 32, in run_phase run(project, workspace, args.stage) python2. 7/dist- packages/ mojo/phase. py", line 716, in run output = "{}\n{} ".format( e.output, dl.get_ log_output( )) python2. 7/dist- packages/ mojo/juju/ debuglogs. py", line 64, in get_log_output logs(): python2. 7/dist- packages/ mojo/juju/ debuglogs. py", line 116, in _gather_logs remote_ command( unit, command, sanitized_ path_list) python2. 7/dist- packages/ mojo/juju/ debuglogs. py", line 101, in _run_remote_command check_output( cmd, stderr= subprocess. STDOUT) python2. 7/subprocess. py", line 566, in check_output python2. 7/subprocess. py", line 710, in __init__ python2. 7/subprocess. py", line 1327, in _execute_child
2016-02-29 21:14:57 [INFO] Using pre-existing container class lxc
2016-02-29 21:14:57 [INFO] Pulling secrets from /srv/mojo/
2016-02-29 21:14:57 [INFO] Running script verify-nagios
2016-02-29 21:21:46 [DEBUG] Debug Log Config found: {'service-cache': {'logs': ['/srv/
2016-02-29 21:22:17 [ERROR] Unknown error
Traceback (most recent call last):
File "/usr/lib/
args.func(args)
File "/usr/lib/
return method(*args, **kwargs)
File "/usr/lib/
phase.
File "/usr/lib/
exception_
File "/usr/lib/
for log in self._gather_
File "/usr/lib/
content = self._run_
File "/usr/lib/
return subprocess.
File "/usr/lib/
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib/
errread, errwrite)
File "/usr/lib/
raise child_exception
OSError: [Errno 7] Argument list too long