retracer requires a lot of storage for instance sandboxes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Daisy |
New
|
High
|
Unassigned |
Bug Description
I'm fairly certain that the cleanup of caches used by the retracers is never done. In setup_cache of retracer.py we can see the following:
So when retracer.py exits the the instance_sandbox directory should be deleted. However, this isn't happening and the sandbox directories are growing quite large. Running the retracer manually, and pressing "Ctrl C" to interrupt the retracer we see "Shutting down." (also using atexit.register) in the retracer log file corresponding to the architecture.
However, running the retracer via the upstart job there is no such "Shutting down" log message leading me to believe that atexit never occurs and this is why the cleanup of caches is not being done regularly.
Related branches
- Evan (community): Approve
-
Diff: 15 lines (+5/-0)1 file modifieddaisy/retracer.py (+5/-0)
Changed in daisy: | |
importance: | Undecided → High |
summary: |
- cleanup of instance sandboxes is never done + retracer requires a lot of storage or instance sandboxes |
Actually, it seems more likely that the retracers are taking too long to exit and so then they get killed. From the cookbook:
Upstart waits for up to kill timeout seconds (default 5 seconds) for the process to end.
If the process is still running after the timeout, a SIGKILL signal is sent to the process which cannot be ignored and will forcibly stop the processes in the process group.
So perhaps we should increase the kill timeout for the upstart jobs.