byobu crashes on start 'pipe: too many open files'

Reported by AJRay on 2011-02-06
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
byobu
Medium
Dustin Kirkland 
byobu (Ubuntu)
Medium
Dustin Kirkland 

Bug Description

Logging into a shared server with multiple users, running byobu crashes (bringing the terminal with it.

It appears to start up partially (showing the two lines at the bottom). However what the lines display is:
first line:
-* ajray@short 152.14.93.72
second line:
pipe: Too many open files unavailable

I can't exit/suspend/interact with byobu, effectively crashing that terminal (forcing me to start a new one.

I can verify that just running screen normally works, and that i can start multiple screens without running into the 'pipe: Too many open files' error.

Let me know what else I can supply to help
~alex

Dustin Kirkland  (kirkland) wrote :

Hi there, thanks for the report.

Well that is a bummer!

Is this a very underpowered machine? How cpu/ram does it have?

Can you try this? Can you disable all of the status notifications? Just go through the variables in ~/.byobu/status and set them all to 0, and then try to run byobu again...

Changed in byobu:
importance: Undecided → High
status: New → Incomplete
zorun (zorun-42) wrote :

I've been able to reproduce this bug, also on a shared server providing shells for a lot of users.

It looks like an agressive setup regarding user limits: attached is the output of `ulimit -a' on such a server.
Particulary interesting is the 'max user processes' which is set to 24; since byobu is written in bash, it means a lot of forks (and I guess each status notification runs in a separate process).

Changed in byobu:
status: Incomplete → Triaged
Dustin Kirkland  (kirkland) wrote :

Zorun,

Hiya, thanks for the information!

Can you tell me what version of Byobu you were running when you reproduced this?

Scott Moser recently did some work, removing hundreds of shell forks from Byobu. We did some performance testing and Byobu's performance improved several times over.

Could you try to perhaps reproduce this with the latest Byobu? I'll also test it in a VM and load that VM with lots of users. Thanks!!!

Dustin Kirkland  (kirkland) wrote :

So, FYI, I ran this here on my Ubuntu 11.04 desktop and I see that I have unlimited user processes by default:

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Dustin Kirkland  (kirkland) wrote :

ajray,

Could you please paste the output of "ulimit -a" on your affected server?

Thanks!
:-Dustin

Changed in byobu:
status: Triaged → Incomplete
zorun (zorun-42) wrote :

@Dustin: I actually tried this a while ago, with something like 3.10

Just tried with the latest 4.9 version, and I still get the same thing... It creates the basic screen structure, but then it just hangs by saying 'pipe: Trop de fichiers ouverts' (too many files open) in the status bar.

I tried setting everything to 0 in my ~/.byobu/status, and the result is exactly the same...
(by the way, I had nothing else running under my user account, excepted the shell in which I launched byobu)

As for the limitation of the number of processes, on any linux box, it is unlimited. In the case of a shared server, such a limit makes sense as it prevents a user to launch too many programs, and thus overloading the server to the detriment of other users.

Hope this helps ;)

Dustin Kirkland  (kirkland) wrote :

Great, thanks for the information. Marking back to triaged.

I'll take a bit closer look at it, and see if there's anything else we can do. Worst case, I'll have byobu check the ulimit on the number of processes and gracefully fail back to a byobu mode with no status notifications enabled.

Changed in byobu:
status: Incomplete → Triaged
Dustin Kirkland  (kirkland) wrote :

Reproduced with:

guest@x201:~$ ulimit -n 12
guest@x201:~$ byobu
.: 39: 3: Too many open files

Dustin Kirkland  (kirkland) wrote :

I added an entry to the manpage saying:

+Byobu requires a suitable \fBulimit\fP(3) values to run. If you get an error at startup saying, 'pipe: too many open files', then check your ulimit -a values, as your "open files" or "max user processes" are too low. In this case, you will probably need to run simple \fBscreen\fP(1)

Changed in byobu (Ubuntu):
status: New → Fix Committed
Changed in byobu:
status: Triaged → Fix Committed
Changed in byobu (Ubuntu):
importance: Undecided → Medium
Changed in byobu:
importance: High → Medium
assignee: nobody → Dustin Kirkland (kirkland)
Changed in byobu (Ubuntu):
assignee: nobody → Dustin Kirkland (kirkland)
Dustin Kirkland  (kirkland) wrote :

I think that's the best I can do for now.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package byobu - 4.43-0ubuntu1

---------------
byobu (4.43-0ubuntu1) precise; urgency=low

  [ Dustin Kirkland ]
  * usr/bin/byobu, usr/lib/byobu/ip_address, usr/share/man/man1/byobu.1:
    - support IP_EXTERNAL fetching, LP: #873736
    - put ip address in the terminal title bar, since we ran out of room
      for it in tmux
  * usr/bin/byobu-select-session: LP: #869390
    - fix select session to obey BYOBU_BACKEND
  * configure.ac, usr/bin/Makefile.am: LP: #869377
    - provide byobu-tmux and byobu-screen symlinks
  * configure.ac, README: LP: #861291
    - configure should not change files under vcs control
    - compiling users need to ensure that BYOBU_PREFIX is set in
      their environment
  * usr/bin/byobu-launcher-install, usr/bin/byobu-launcher-uninstall:
    - LP: #861534
    - ensure launcher command removed from each file before installing
  * usr/lib/byobu/.dirs:
    - use /dev/shm for cache, LP: #872551, #869279
    - search and use a shared dir when possible
  * usr/lib/byobu/battery: LP: #851100
    - rework the battery logic to use /sys instead of /proc
  * usr/share/man/man1/byobu.1: LP: #713879
    - note too many files open ulimit error in documentation

  [ Daniel Hahler ]
  * usr/bin/byobu, usr/bin/byobu-launcher-install, usr/bin/byobu-
    launcher-uninstall: Use BYOBU_PREFIX explicitly, LP: #861302
  * debian/rules, README: LP: #854868
    - refactor autoconf into debian/rules

  [Scott Moser]
  * usr/lib/byobu/.shutil: fix rounding error in fpdiv, LP: #869282
 -- Dustin Kirkland <email address hidden> Fri, 21 Oct 2011 13:43:57 -0500

Changed in byobu (Ubuntu):
status: Fix Committed → Fix Released
Changed in byobu:
status: Fix Committed → Fix Released
Scott Moser (smoser) wrote :

Just as a point of reference:
$ dpkg-query --show byobu
byobu 4.37-0ubuntu1

$ bhome=/home/smoser/t/byobu-home
$ mkdir -p $bhome
$ i=1; while :; do i=$(($i+1)); echo -n = $i =:; \
   HOME=$bhome sh -c "ulimit -n $i && exec byobu true" && break; done
=== 2 ===:/bin/sh: error while loading shared libraries: libc.so.6: cannot open shared object file: Error 24
=== 3 ===:/bin/sh: error while loading shared libraries: libc.so.6: cannot open shared object file: Error 24
=== 4 ===:/bin/sh: 3: Invalid argument
=== 5 ===:/bin/sh: 3: Invalid argument
=== 6 ===:/bin/sh: 3: Invalid argument
=== 7 ===:/bin/sh: 3: Invalid argument
=== 8 ===:/bin/sh: 3: Invalid argument
=== 9 ===:/bin/sh: 3: Invalid argument
=== 10 ===:/bin/sh: 3: Invalid argument
=== 11 ===:.: 41: 3: Too many open files
=== 12 ===:.: 39: 3: Too many open files
=== 13 ===:.: 37: 3: Too many open files
=== 14 ===:=== 15 ===:[screen is terminating]

What the above shows is that 'ulimit -n 15' worked. Anything less than that
was insufficient. The alternate 'HOME' was just to get default byobu configuration. By comparison, 'screen true' will run with 5.

Just information.

Dustin Kirkland  (kirkland) wrote :

Okay, I can put a ulimit check in byobu's startup.

Changed in byobu (Ubuntu):
status: Fix Released → Triaged
Changed in byobu:
status: Fix Released → Triaged
zorun (zorun-42) wrote :

Interesting.

On the machine I've tested it, the maximum number of open files (-n) was 1024, but the issue showed up anyway.

What's more an issue, I guess, is the -u limit ("max user processes") which is set at 24.

Dustin Kirkland  (kirkland) wrote :

Okay, I added a couple of ulimit checks to the byobu -v path. It costs a fork (0.01s) for each of the shell outs to ulimit, which shouldn't be necessary in the default code path. However, I can see this being useful if someone runs 'byobu -v'. Thanks for the information. Ping me if you guys find any other ulimit values/limits worth testing! Cheers!

Changed in byobu:
status: Triaged → In Progress
Changed in byobu (Ubuntu):
status: Triaged → In Progress
Changed in byobu:
status: In Progress → Fix Committed
Changed in byobu (Ubuntu):
status: In Progress → Fix Committed
Changed in byobu:
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package byobu - 4.46-0ubuntu1

---------------
byobu (4.46-0ubuntu1) precise; urgency=low

  * debian/control: LP: #887344
    - promote tmux to a 'recommends'; will 'depend' on it soon
  * === added directory etc/profile.d, etc/byobu/Makefile.am,
    etc/profile.d/Makefile.am, etc/profile.d/Z97-byobu.sh, Makefile.am:
    - LP: #586546
    - Enable any user to remotely opt into launching byobu by default,
      using an LC_BYOBU=1 environment variable, which is transmitted over
      SSH
  * configure.ac, debian/install, etc/byobu/Makefile.am,
    etc/profile.d/Makefile.am:
    - fix installations to /etc
  * usr/bin/byobu, usr/bin/byobu-reconnect-sockets,
    usr/share/byobu/profiles/tmux: LP: #883637
    - fix reconnection to SESSION_MANAGER and DBUS_SESSION_BUS_ADDRESS
    - add C-F5 key to tmux profile
  * usr/bin/byobu-ctrl-a, usr/share/byobu/profiles/tmux: LP: #887387
    - support F12 as escape key in tmux mode
    - reorder byobu-ctrl-a to offer screen first, then emacs mode
    - make byobu-ctrl-a slighty more compatible with tmux (not quite
      there yet...)
  * usr/bin/byobu: LP: #713879
    - add ulimit checks to byobu -v
  * usr/lib/byobu/.constants:
    - don't use UTF8 C an F for now
  * usr/lib/byobu/network: LP: #880410
    - localize some variables, fix variable colision with cpu_temp
  * debian/control:
    - bump standards version
  * debian/install, debian/rules, etc/byobu/Makefile.am,
    etc/profile.d/Makefile.am:
    - fighting with autoconf to get shtuff installing in /etc, argh
 -- Dustin Kirkland <email address hidden> Mon, 31 Oct 2011 09:46:46 -0400

Changed in byobu (Ubuntu):
status: Fix Committed → Fix Released

I used byobu 4.37-0ubuntu1 in Ubuntu Server 11.10 64 bit.

I left byobu running. But every few hours it will automatically close saying

"pipe: Too many open files y".

Ulimit -n was 1024.

I upgraded to 5.16-0ubuntu1~ppa5 and the problem still existed. I raised ulimit -n to 4096. Instead of crashing it will go into an infinite loop using 100% CPU.

These were 2 backtraces from the infinite loop.

#0 0x00007f7ac5ad25f3 in select () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00000000004447bc in ?? ()
#2 0x0000000000404e3f in ?? ()
#3 0x00007f7ac5a1730d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00000000004057a9 in ?? ()
#5 0x00007fffe7cad3f8 in ?? ()
#6 0x000000000000001c in ?? ()
#7 0x0000000000000006 in ?? ()
#8 0x00007fffe7cad7f6 in ?? ()
#9 0x00007fffe7cad7fd in ?? ()
#10 0x00007fffe7cad800 in ?? ()
#11 0x00007fffe7cad806 in ?? ()
#12 0x00007fffe7cad809 in ?? ()
#13 0x00007fffe7cad82c in ?? ()
#14 0x0000000000000000 in ?? ()

#0 0x00007f7ac5aef8b0 in __read_chk () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x000000000041ec7e in ?? ()
#2 0x0000000000444853 in ?? ()
#3 0x0000000000404e3f in ?? ()
#4 0x00007f7ac5a1730d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00000000004057a9 in ?? ()
#6 0x00007fffe7cad3f8 in ?? ()
#7 0x000000000000001c in ?? ()
#8 0x0000000000000006 in ?? ()
#9 0x00007fffe7cad7f6 in ?? ()
#10 0x00007fffe7cad7fd in ?? ()
#11 0x00007fffe7cad800 in ?? ()
#12 0x00007fffe7cad806 in ?? ()
#13 0x00007fffe7cad809 in ?? ()
#14 0x00007fffe7cad82c in ?? ()
#15 0x0000000000000000 in ?? ()

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments