Terminator has a (some?) memory leak

[Simon Déziel]

Binary package hint: terminator

I've been running terminator for more than a week (I suspend/resume my laptop everyday) and now terminator memory usage is very high. top output :

21557 simon 20 0 602m 177m 18m S 0 2.3 18:57.39 /usr/bin/python /usr/bin/terminator

I currently have 2 tabs open, the first being divided in 8 and the other in 3. When I try subdivide one of the sections, I get this error :

Unable to start shell:/bin/bash

I am probably hitting a ulimit or something related. Additional info :

$ uptime
 22:05:15 up 9 days, 10:17, 23 users, load average: 1.15, 1.08, 1.12

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

$ lsb_release -rd
Description: Ubuntu 11.04
Release: 11.04

$ apt-cache policy terminator
terminator:
  Installed: 0.95-1
  Candidate: 0.95-1
  Version table:
 *** 0.95-1 0
        500 http://ca.archive.ubuntu.com/ubuntu/ natty/universe amd64 Packages
        100 /var/lib/dpkg/status

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: terminator 0.95-1
ProcVersionSignature: Ubuntu 2.6.38-9.43-generic 2.6.38.4
Uname: Linux 2.6.38-9-generic x86_64
Architecture: amd64
Date: Thu May 19 22:00:13 2011
PackageArchitecture: all
ProcEnviron:
 LANGUAGE=en_CA:en
 LANG=en_US.UTF-8
 LC_MESSAGES=en_CA.UTF-8
 SHELL=/bin/bash
SourcePackage: terminator
UpgradeStatus: No upgrade log present (probably fresh install)

[Simon Déziel (sdeziel)]

I just closed all but one of the division and the memory usage doesn't drop. Top output :

21557 simon 20 0 604m 179m 18m S 0 2.3 19:10.68 /usr/bin/python /usr/bin/terminator

[Simon Déziel (sdeziel)]

On my system, a freshly started terminator gives this kind of top output :

  624 simon 20 0 431m 35m 16m S 1 0.5 0:00.51 /usr/bin/python /usr/bin/terminator

[Andrea Corbellini (andrea.corbellini)]

AFAIK, terminator is pure-Python. So I do not think that it's leaking really memory. Instead, I think that it's creating some kind of temporary object without deleting them (probably because of circular references).

I'll try to do some debugging this week or the next one. If anyone is interested, checking for circular references is simple: drop into pdb and "import gc; gc.collect(); gc.garbage".

[Andrea Corbellini (andrea.corbellini)]

Some debug revealed that whenever a terminal is split (or a new window is created, Terminator creates some Python objects, but never deletes them when the terminals are closed. So there are no memory leaks nor circular references.

[Chris Jones (cmsj)]

Andrea: which objects are you seeing that happening for?

[Simon Déziel (sdeziel)]

Just to mention that its still occurring with terminator from Precise:

 5132 simon 20 0 868m 184m 18m S 2 2.4 9:57.44 /usr/bin/python /usr/bin/terminator

$ apt-cache policy terminator
terminator:
  Installed: 0.96-0ubuntu1
  Candidate: 0.96-0ubuntu1
  Version table:
 *** 0.96-0ubuntu1 0
        500 http://archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages
        100 /var/lib/dpkg/status

[Stephen Boddy (stephen-j-boddy)]

Hi Bug subscribers, this is the maintainer (Steve Boddy) of Terminator, who took over from Chris.

In case you were not aware there has been a new release 0.98 of Terminator in the last few days which has many changes, fixes and updates. We also have an ongoing GTK3 port which uses a much more up-to-date libvte.

Even with the ~50 issues closed by that release, and discounting the ~90 outstanding wishlist items, we still have an unwieldy 95 bugs. With the oldest approaching it's 6th birthday and some that haven't seen any updates in nearly 5, it is time for a purge.

My intention is to try and get this list down to a minimum, and for that I need your help.

First I'm going to work through, oldest to newest, setting bugs to Incomplete, pending confirmation that they are still an issue. If after a month there is no feedback, the issue will get closed as Invalid. Here's where you come in. I can't always reproduce, or setting up (i.e. other distros) to even test takes a long time, so I'm going to try crowdsourcing this.

1. Get the latest release. Is it still an issue? If yes, set the confirmed-0.98 tag. For bonus points follow the guide http://gnometerminator.blogspot.com/2015/09/so-you-want-to-try-terminator-gtk3.html and see if the GTK3 branch has same issue, and set the confirmed-gtk3 or notaffected-gtk3 tag.

2. Can you reproduce with an empty/default config file? You can pass "-g temp_config" to do this without affecting your existing config. If you already have Terminator running, you'll need to pass -u as well to disable the DBus.

3. If able to, can you reproduce with another user/guest account?

4. If the issue does not already have clear, precise instructions to reproduce with minimal setup/steps, add them.

5. Put as much info about what you are running.
   * Software/library versions (libvte, gtk, pango, cairo, distro, desktop environment, language)
   * Screens (single or multi)
   * Any unusual / custom packages or configs that might be interacting.

6. If possible attach the config file, unless the issue also happens with an empty one, but mention that.

7. Attach the "-d" debug output from 0.98.

Some common things that cause incidents.

* Strange sizing issues or rapidly shrinking windows - Try turning off Window geometry hints.
* Input (esp. broadcasting duplicate chars etc) problems - Try killing IBus.
* Hand editing of the config file can cause various issues - Due to misunderstanding, incorrect structure or typos.
* Some issues are actually in libvte - This is why testing GTK3 is so important.
* segfaults - Unfortunately these are dying inside the C libraries, and it is usually beyond me to fix those.

Many thanks for your assistance, and hopefully this will get us closer to a bug-free Terminator.

[Daye Lao (bid4bidden)]

I happened to hit the same problem and so did some digging. lsof output showed that terminator's (grand-)process gdm is leaking file descriptor:

/usr/bin/ 2143 ldaye 25u REG 8,3 6413711 42205304 /tmp/vteZRV4BY (deleted)
/usr/bin/ 2143 ldaye 26u REG 8,3 1070400 42205305 /tmp/vteHMV4BY (deleted)
.
.
.
/usr/bin/ 2143 ldaye 1020u REG 8,3 2407 42206629 /tmp/vte7G4CFY (deleted)
/usr/bin/ 2143 ldaye 1021u REG 8,3 113 42206606 /tmp/vteY9YUFY (deleted)
/usr/bin/ 2143 ldaye 1022u REG 8,3 64 42206627 /tmp/vteO5YUFY (deleted)
/usr/bin/ 2143 ldaye 1023u REG 8,3 3350 42206628 /tmp/vteTCOLFY (deleted)

Also, strace on gdm showed that open() failed with EMFILE (Too many open files), which matches well with the limit of open files is at 1024.

Web search on "gdm file descriptor leak" landed on a few hits, the following could be relevant:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=532319

[Daye Lao (bid4bidden)]

I misread the bug in the link in the previous comment, which seems to be unrelated. However the following issue does look the same to the issue at hand:

https://bugzilla.redhat.com/show_bug.cgi?id=647942

[Stephen Boddy (stephen-j-boddy)]

Hi, you're the first person to touch this bug in a verrrrrrry looooooooong time. Which version of Terminator are you using?

Quick explanation: The /tmp/vteXXXXXX files are temporary files created to hold the scrollback history once the size gets a little too big. You will typically see these if you have infinite scrollback turned on. They are created, a lock is obtained, then it is unlinked so no longer appears in the filesystem. They are entirely handled by the libvte component, so nothing can be done within Terminator directly. The vte developers have long since moved on from the crusty old libvte used by Terminator (typically 0.28) to a much newer gtk3 version (currently 0.4x), and do not backport fixes. It would help if you could see if this is still an issue in the gtk3 version of Terminator with a recent version of libvte. ( http://gnometerminator.blogspot.de/2015/09/so-you-want-to-try-terminator-gtk3.html ) If the problem still occurs there, then a bug needs to be raised against libvte ( https://bugzilla.gnome.org/page.cgi?id=browse.html&product=vte )

[Sparhawk (sparhawkthesecond)]

I've hit this bug recently too. More information here: https://bbs.archlinux.org/viewtopic.php?id=213319

It happens every few days. It's almost certainly Terminator to blame and not something more systemic, because I can open Konsole fine. There may be a memory leak involved. A freshly started terminator uses 49 MB, whereas the stuck version uses 190 MB.

Unfortunately, this bug takes a while to surface. This means testing terminator-gtk3 is not an option for me; version r1637 is unusable because of this bug: https://bugs.launchpad.net/terminator/+bug/1501128/comments/4

I'm happy to troubleshoot more, if that is useful.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Stephen Boddy (stephen-j-boddy)]

Committed revision 1729.

Yay!!! This was an annoying one, and it seems it has been around for a long time.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Dependencies.txt" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Vej (vej)]

@rahmadani

You assigned this bug report to yourself, which has been of status "Fix Released" in Terminator before. Can you explain please, why you consider this to be in need of another fix, or what kind of work you are going to do here?

[Stephen Boddy (stephen-j-boddy)]

@Vej: I've had a few cases of people in Indonesia doing stupid actions that cause me (the maintainer) problems, as I have to then fix the stuff they mess with. I had one Indonesian person that I had banned twice for doing braindead changes to the translations. I do wonder if this crap is some way of accumulating karma, but why? It's not like there's cash rewards for a big karma score. And why is it always Indonesia? Very peculiar.

@rahmadani: This is the first and last warning to you. If you make more pointless changes to this project then I will speak to the Launchpad group and have your account banned. In case you do not speak English: Ini adalah peringatan pertama dan terakhir untuk Anda. Jika Anda membuat perubahan yang lebih sia-sia untuk proyek ini maka saya akan berbicara kepada kelompok Launchpad dan memiliki account Anda dilarang.

[Vej (vej)]

@Stephen: Thanks for your explanation.

@rahmadani: Unassigning you until you come back to us with the requested informations.