DoS by spawning hanging byobu-status processes

[Przemek Wesolek]

When I run byobu-tmux, after a minute or so the kernel reports:

    bash: fork: retry: Resource temporarily unavailable

and nothing more than finishing byobu can be done (as no command can be run).

Looking at the process list reveals something like:

myuser 27821 12220 0 06:14 pts/0 00:00:00 tmux -2 -f /opt/byobu/share/byobu/profiles/tmuxrc new-session /opt/byobu/bin/byobu-shell
myuser 27842 1 0 06:14 ? 00:00:00 tmux -2 -f /opt/byobu/share/byobu/profiles/tmuxrc new-session /opt/byobu/bin/byobu-shell
myuser 27843 27842 0 06:14 /root/dev/pts/1 00:00:00 /bin/bash
myuser 27845 27842 0 06:14 ? 00:00:00 /bin/sh /opt/byobu/bin/byobu-status tmux_left
myuser 27846 27842 0 06:14 ? 00:00:00 /bin/sh /opt/byobu/bin/byobu-status tmux_right
myuser 27852 27845 0 06:14 ? 00:00:00 tmux set -g status-left-length 52
myuser 27857 27846 0 06:14 ? 00:00:00 tmux set -g status-left-length 52
myuser 27896 27842 0 06:14 ? 00:00:00 /bin/sh /opt/byobu/bin/byobu-status tmux_left
myuser 27897 27842 0 06:14 ? 00:00:00 /bin/sh /opt/byobu/bin/byobu-status tmux_right
myuser 27900 27897 0 06:14 ? 00:00:00 tmux set -g status-left-length 52
myuser 27901 27896 0 06:14 ? 00:00:00 tmux set -g status-left-length 52
myuser 27949 27842 0 06:14 ? 00:00:00 /bin/sh /opt/byobu/bin/byobu-status tmux_left
myuser 27950 27842 0 06:14 ? 00:00:00 /bin/sh /opt/byobu/bin/byobu-status tmux_right
myuser 27953 27949 0 06:14 ? 00:00:00 tmux set -g status-left-length 52
myuser 27954 27950 0 06:14 ? 00:00:00 tmux set -g status-left-length 52
myuser 27971 27842 0 06:14 ? 00:00:00 /bin/sh /opt/byobu/bin/byobu-status tmux_left
myuser 27972 27842 0 06:14 ? 00:00:00 /bin/sh /opt/byobu/bin/byobu-status tmux_right
myuser 27975 27972 0 06:14 ? 00:00:00 tmux set -g status-left-length 52
myuser 27976 27971 0 06:14 ? 00:00:00 tmux set -g status-left-length 52

as so on with "byobu-status" and "tmux set" processes being added every second.

Also, as a result, the status line is not being displayed.

I checked every released version since 4.55, and the problem appears only in 5.18.

Tested under bluehost shared server
byobu: 5.18
tmux: 1.6
bash: 4.1.2(1)-release (x86_64-redhat-linux-gnu)

[Dustin Kirkland  (kirkland)]

Hi, thanks for the report.

I've seen this from time to time on heavily multi-user, shared servers such as bluehost.

Can you try backing your status interval off? The default is *very* aggressive (1 second). Perhaps you need something a little more generous in your environment.

Try adding:
 set -g status-interval 15
to ~/.byobu/.tmux.conf

[Przemek Wesolek (pwes)]

Unfortunately, increasing status refresh interval only postpones the problems, as the processes are still running in the background. There is command running:

    /bin/sh /home6/wesolekn/opt/byobu/bin/byobu-status tmux_left

which is waiting for a child to fininsh:

    tmux set -g status-left-length 52

strace-ing that 'tmux set' command shows it hanging on

    epoll_wait(3,

where the descriptor 3 points to

    anon_inode:[eventpoll]

As a result, no status bar is displayed and the list of processes is growing.

Because the bug is pretty well locatable (doesn't show up in 5.17, occurs in 5.18), maybe I could bisect the commits to locate the exact cause? Could you please show me a short script to do so, as I don't have bazaar experience.

[Przemek Wesolek (pwes)]

After digging more, it looks like a regression from #997766. When I comment the lines changed in revision 1942, byobu and tmux works correctly, the status line refreshes and no processes hang in the background.

Finally, I suspect some race condition. Here is why. After first "hanging" status-update, I exit byobu. Then, 'ps' shows the following hanging:

user 22803 0.0 0.0 106212 1424 ? S 08:08 0:00 /bin/sh /home6/user/opt/byobu/bin/byobu-status tmux_left
user 22804 0.0 0.0 106212 1428 ? S 08:08 0:00 /bin/sh /home6/user/opt/byobu/bin/byobu-status tmux_right
user 22806 0.0 0.0 23760 1168 ? S 08:08 0:00 tmux set -g status-left-length 52
user 22809 0.0 0.0 23760 1168 ? S 08:08 0:00 tmux set -g status-left-length 52

So, two 'tmux set' hang. Let's kill one of them (22809) and the list looks like this

user 22803 0.0 0.0 106212 1424 ? S 08:08 0:00 /bin/sh /home6/user/opt/byobu/bin/byobu-status tmux_left
user 22804 0.0 0.0 106212 1428 ? S 08:08 0:00 /bin/sh /home6/user/opt/byobu/bin/byobu-status tmux_right
user 22806 0.0 0.0 23760 1168 ? S 08:08 0:00 tmux set -g status-left-length 52
user 23731 0.0 0.0 23760 1168 ? S 08:09 0:00 tmux set -g status-right-length 156

Now, it hangs on setting the right length. Let's kill that too (23731):

user 22803 0.0 0.0 106212 1424 ? S 08:08 0:00 /bin/sh /home6/user/opt/byobu/bin/byobu-status tmux_left
user 22806 0.0 0.0 23760 1168 ? S 08:08 0:00 tmux set -g status-left-length 52

Unfortunatelly, the first left status update process still hangs. I kill 22806, and then status-right-length, on which there was a hang previously doesn't show, all update processes finish correctly.

[Jonas Jelten (jonas-jelten)]

My server also gets flooded with tmux processes, but it works just as it should.

3 game server account currently have 480 tmux processes running, they once used 2k processes.

I really can't imagine that this amount of processes is neccessary.

[Neil Perry (nperry)]

I believe this is now confirmed and we can see the regression.

[Daniel Hahler (blueyed)]

For what it's worth: I am experiencing this on my Synology Diskstation DS212+ (armv5tel), but not on my notebook, which is more powerful.

A few observations:
 - "read < file" returns with exit code 1 on zsh, if there is no newline in the file (and therefore not a whole "line" has been read). This has to be changed, either by adding a newline to the printf or by comparing the read value instead of setting it to 0 always in that case.
 - there should be locking, either around the "tmux set" calls, or even better around the calls to byobu-status.
 - the adjustment of width should probably only happen with $1=tmux_left, and not for tmux_right at nearly the same time. This might be true for most of the byobu-status script: with tmux, it gets called twice nearly at the same time, with different argument.

I am pasting the diff from me fiddling around with this code, for inspiration:

diff --git i/lib/byobu/usr/bin/byobu-status w/lib/byobu/usr/bin/byobu-status
index b3bec70..0e76660 100755
--- i/lib/byobu/usr/bin/byobu-status
+++ w/lib/byobu/usr/bin/byobu-status
@@ -36,17 +36,26 @@ for i in "${BYOBU_PREFIX}/share/$PKG/status/status" "${BYOBU_PREFIX}/share/$PKG/
 done

 # Fix status printing for small terminal sizes
+if [ "$1" = "tmux_left" ]; then
 width=$(tmux list-windows -F "#{session_width}")
+if [ ! -e "$BYOBU_RUN_DIR/lock.set-width" ]; then
+ touch "$BYOBU_RUN_DIR/lock.set-width"
+ date >> /tmp/byobu.log
 w_last=0
 [ -r "$BYOBU_RUN_DIR/width" ] && read w_last < $BYOBU_RUN_DIR/width 2>/dev/null 1>&2 || w_last=0
 for w in $width; do
+ echo $w / $w_last >> /tmp/byobu.log
        if [ "$w" != "$w_last" ]; then
                tmux set -g status-left-length $(($w*1/4)) >/dev/null 2>&1
+ sleep 5
                tmux set -g status-right-length $(($w*3/4)) >/dev/null 2>&1
- printf "$w" > $BYOBU_RUN_DIR/width
+ printf "$w\n" > $BYOBU_RUN_DIR/width # newline is important for read in zsh
                break
        fi
 done
+ rm "$BYOBU_RUN_DIR/lock.set-width"
+fi
+fi

 case "$BYOBU_BACKEND" in
        screen)

There is still a hangup of the "tmux set" call, but at least it does not pile up anymore and the status gets updated after all.

[Christoph Lange (allegristas)]

I can confirm this with Byobu 5.18 and tmux 1.6. At the moment, while I'm new to Byobu and occasionally ending my Byobu session to switch back to tmux or to an unmanaged terminal, this is particularly annoying, because the tmux processes that keep running after existing Byobu (exiting, not detaching!) make it impossible to start Byobu again; it will simply exit immediately. "pstree", run from somewhere else, shows tmux and some leftover byobu-status processes:

tmux───2*[byobu-status───tmux]

These are fewer the higher the status-interval is, but still…

"tmux ls", run from somewhere else, however, no longer shows tmux sessions that one could attach to, so in fact there are some tmux zombies hanging around.

[Nicolas (nicolas-hoeft)]

Daniel, your locking mechanism is actually very bad, this will eventually break.
Do not use non-atomic operations for locking!

A better approach for locking is actually something like flock

[Daniel Hahler (blueyed)]

Is anything being worked on in this regard?

Would a locking mechanism like proposed by me (see above), using flock instead, make sense?

[yorkfinechan (yorkfinechan)]

I am not sure that my bug is like this. When I start byobu the byobu-status tmux_left and byobu-status tmux_right processes increase infinitely, even when I quit byobu, there is still a tmux process and lots of byobu-status process

I look into the byobu-status source and add some echo, finally it shows me that "tmux set -g status-left-length $(($w*1/4)) >/dev/null 2>&1" hang on

it seems to be a tmux's bug
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=631984

I am not able to solve this problem, so I follows the instruction from Nicholas Marriott

tmux set -g status-left-length $(($w*1/4)) | cat >/dev/null 2>&1
tmux set -g status-right-length $(($w*3/4)) | cat >/dev/null 2>&1

just use cat but not redirect into /dev/null directly.

byobu works correctly for me....

sorry, I am not good at English

[Przemek Wesolek (pwes)]

Good catch, yorkfinechan!

I tested your solution, works for me too.

Linked a branch lp:~pwes/byobu/1003938 with a fix.

[Alessandro Tagliapietra (tagliapietra.alessandro)]

Hi guys, my pstree shows

0 S root 4663 2422 0 80 0 - 1100 wait 616 7 10:17 ? 00:00:00 sh -c byobu-status tmux_right
0 S root 4666 4663 0 80 0 - 1100 wait 840 5 10:17 ? 00:00:00 /bin/sh /usr/bin/byobu-status tmux_right
1 D root 4670 4666 0 80 0 - 1100 stop_o 232 1 10:17 ? 00:00:00 /bin/sh /usr/bin/byobu-status tmux_right

and my load is very high, running byobu 5.17, can be this the same issue?

[Przemek Wesolek (pwes)]

I'm not sure, for me 5.17 worked well, problems started with 5.18.

[Przemek Wesolek (pwes)]

Dustin, is there a chance to see the fix branch merged into the main one?

[Dustin Kirkland  (kirkland)]

Sure! Just merged and committed. Hope that fixes this for you!

Committed revision 2036.

[Alessandro Tagliapietra (tagliapietra.alessandro)]

Which version has this fix? I've 5.33 and my server load is at 150, ps shows lot of byobu-status in D state

[Alessandro Tagliapietra (tagliapietra.alessandro)]

No matter, just talking to you in irc, thanks!

[Smith (smith434)]

I believe I still has this issue, I'm running bybobu 5.41 and tmux 1.8. Did the fix include avoiding the issue for custom notifications?