retracers repeatedly cannot allocate memory

[Brian Murray]

From a log file on a retracer:

2016-05-31 23:35:54,068:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:Processing.
2016-05-31 23:35:54,268:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:Decompressing to /tmp/tmpklv_4u-swift.b3b81708-2779-11e6-9984-fa163ebeb28a.oopsid.core
2016-05-31 23:35:55,884:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:Retracing b3b81708-2779-11e6-9984-fa163ebeb28a:swift
2016-05-31 23:38:40,957:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:Removing /srv/daisy.ubuntu.com/production/cache/Ubuntu 16.04/cache-k7C7iv/sandbox
2016-05-31 23:38:41,023:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:Apport's return code was 1.
2016-05-31 23:38:41,023:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:Installing extra package korganizer to get ExecutablePath
2016-05-31 23:38:41,023:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:Traceback (most recent call last):
2016-05-31 23:38:41,023:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/srv/daisy.ubuntu.com/production/apport/bin/apport-retrace", line 301, in <module>
2016-05-31 23:38:41,024:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: options.dynamic_origins)
2016-05-31 23:38:41,024:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/srv/daisy.ubuntu.com/production/apport/apport/sandboxutils.py", line 228, in make_sandbox
2016-05-31 23:38:41,024:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: architecture=report.get('Architecture'), origins=origins)
2016-05-31 23:38:41,024:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/srv/daisy.ubuntu.com/production/apport/apport/packaging_impl.py", line 1020, in install_packages
2016-05-31 23:38:41,024:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: out = subprocess.check_output(['dpkg-deb', '--show', i.destfile]).decode()
2016-05-31 23:38:41,024:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/usr/lib/python3.2/subprocess.py", line 515, in check_output
2016-05-31 23:38:41,024:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: process = Popen(*popenargs, stdout=PIPE, **kwargs)
2016-05-31 23:38:41,024:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/usr/lib/python3.2/subprocess.py", line 745, in __init__
2016-05-31 23:38:41,024:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: restore_signals, start_new_session)
2016-05-31 23:38:41,024:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/usr/lib/python3.2/subprocess.py", line 1197, in _execute_child
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: restore_signals, start_new_session, preexec_fn)
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:OSError: [Errno 12] Cannot allocate memory
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:Installing extra package korganizer to get ExecutablePath
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:Traceback (most recent call last):
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/srv/daisy.ubuntu.com/production/apport/bin/apport-retrace", line 301, in <module>
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: options.dynamic_origins)
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/srv/daisy.ubuntu.com/production/apport/apport/sandboxutils.py", line 228, in make_sandbox
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: architecture=report.get('Architecture'), origins=origins)
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/srv/daisy.ubuntu.com/production/apport/apport/packaging_impl.py", line 1020, in install_packages
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: out = subprocess.check_output(['dpkg-deb', '--show', i.destfile]).decode()
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/usr/lib/python3.2/subprocess.py", line 515, in check_output
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: process = Popen(*popenargs, stdout=PIPE, **kwargs)
2016-05-31 23:38:41,025:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/usr/lib/python3.2/subprocess.py", line 745, in __init__
2016-05-31 23:38:41,026:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: restore_signals, start_new_session)
2016-05-31 23:38:41,026:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: File "/usr/lib/python3.2/subprocess.py", line 1197, in _execute_child
2016-05-31 23:38:41,026:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift: restore_signals, start_new_session, preexec_fn)
2016-05-31 23:38:41,026:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:OSError: [Errno 12] Cannot allocate memory
2016-05-31 23:38:41,032:20177:139696962541312:INFO:root:b3b81708-2779-11e6-9984-fa163ebeb28a:swift:Retrace failed (1), moving to failed queue.
2016-05-31 23:38:41,113:20177:139696962541312:INFO:root:Shutting down.
2016-05-31 23:38:41,616:3172:140530383775488:INFO:root:Running revision number: 708 with sandbox_dir /srv/daisy.ubuntu.com/production/cache, gdb 7.10.90.20160215-0ubuntu3~~0.IS.12.04.0.
Traceback (most recent call last):
  File "/srv/daisy.ubuntu.com/production/daisy/daisy/retracer.py", line 1354, in <module>
    main()
  File "/srv/daisy.ubuntu.com/production/daisy/daisy/retracer.py", line 1339, in main
    failed=options.failed)
  File "/srv/daisy.ubuntu.com/production/daisy/daisy/retracer.py", line 143, in __init__
    universal_newlines=True)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1143, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
2016-05-31 23:38:42,863:3172:140530383775488:INFO:root:Shutting down.
Traceback (most recent call last):
  File "/srv/daisy.ubuntu.com/production/daisy/daisy/retracer.py", line 1354, in <module>
    main()
  File "/srv/daisy.ubuntu.com/production/daisy/daisy/retracer.py", line 1332, in main
    msg += ", gdb %s." % check_output(["dpkg-query", "--show", "-f${Version}", "gdb"]).decode()
  File "/usr/lib/python2.7/subprocess.py", line 537, in check_output
    process = Popen(stdout=PIPE, *popenargs, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1143, in _execute_child
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
2016-05-31 23:38:43,368:3178:139663877138176:INFO:root:Shutting down.

[Brian Murray (brian-murray)]

I discovered multiple retracers that were down this morning and their log files eneded with ImpotErrors and allocating memory after the OSErrors. An example:

Traceback (most recent call last):
  File "/srv/daisy.ubuntu.com/production/daisy/daisy/retracer.py", line 1354, in <module>
    main()
  File "/srv/daisy.ubuntu.com/production/daisy/daisy/retracer.py", line 1319, in main
    retracer_oops_cfg = oops_dictconfig.config_from_dict(config.oops_config)
  File "/usr/lib/pymodules/python2.7/oops_dictconfig/dictconfig.py", line 98, in config_from_dict
    publish_method = publisher_factory(publisher_defn)
  File "/usr/lib/pymodules/python2.7/oops_dictconfig/dictconfig.py", line 45, in _datedir_repo_from_defn
    from oops_datedir_repo import DateDirRepo
  File "/usr/lib/python2.7/dist-packages/oops_datedir_repo/__init__.py", line 34, in <module>
    from oops_datedir_repo.repository import DateDirRepo
  File "/usr/lib/python2.7/dist-packages/oops_datedir_repo/repository.py", line 35, in <module>
    import serializer
  File "/usr/lib/python2.7/dist-packages/oops_datedir_repo/serializer.py", line 36, in <module>
    import bz2
ImportError: /usr/lib/python2.7/lib-dynload/bz2.so: failed to map segment from shared object: Cannot allocate memory
2016-05-31 18:57:22,068:11939:140016601495296:INFO:root:Shutting down.