Corrupt display for history search in vi-mode, 256-color prompt

[Micah Cowan]

Binary package hint: bash

This is on Ubuntu 10.10. It very likely still remains in Natty.

I use bash with a prompt that is derived from the current backgrounded jobs (the script I use to do this is at http://micah.cowan.name/hg/promptjobs/ - you just source the file and it does everything). I've customized the colors used, to take advantage of 256-color support in gnome-terminal. An instance of the prompt that might be produced is:

PS1="\[\033[m\017\033[38;5;103m\]micah-acer\[\033[1m\033[38;5;19m\](\[\033[m\017\033[38;5;83m\]\061\[\033[m\017\033[38;5;189m\]wtitle\[\033[1m\033[38;5;19m\])\[\033[m\017\033[m\017\033[38;5;103m\]$ \[\033[m\017\]"

(This prompt will only display correctly in xterm-compatible terminals; the term I use is gnome-terminal.)

With a prompt like this, and in vi-mode ("set -o vi" in bash), attempting to initiate a search in the history, results in display glitches (specifically, the history line bash/readline jumps to is displayed far over to the right, and with a couple garbage characters before it).

Steps to reproduce:

1. Be in vi-mode ("set -o vi" in bash): in particular, readline's "non-incremental-reverse-search-history" MUST be bound to the "/" key, as this has significant effects on how bash/readline choose to prompt for a history search string (even though, for me at least, the "bind" command doesn't seem to reflect this). If you're running these steps, it would be advisible for you to be sufficiently familiar with vi-style bindings to know how to enter commands.
2. Set PS1 as described above.
3. Invoke the non-incremental-reverse-search-history function by pressing ESC (to escape vi's insert-mode) and "/" (to prompt for reverse history).
4. At this point, the "/" you just typed may not be showing up properly: this is the first symptom that something's wrong.
5. Type in some string that should be present in your recent bash history (so that bash will jump to a different line), and hit enter.

Result:
The history line bash jumps to will be drawn in the wrong location (far right of the prompt), and with garbage characters; typing "k" or "j" (or the cursor keys) to move up or down in history continues to draw these lines in the wrong location.

Expected Result:
The jumped-to line ought to be drawn immediately to the right of the prompt, and without garbage characters before it.

Cause of bug:
This bug is from readline (it is present both in the readline6 source package (and probably older, such as readline5), and bash's own built-in readline code (which has few differences from the sources in readline6). The bug lies in the function rl_message in display.c, which is called by _rl_nsearch_init, which is called from noninc_search. rl_message is primarily intended for writing a "message" on the current line, which doesn't normally include "invisible" characters (escape sequences, like the one I'm using in my prompt to set advanced colors), but in this case is being (ab)used to include the prompt. It uses a buffer of only 128 characters, which in the case of the above prompt/PS1, is overrun. As long as the system library provides vsnprintf, this does not lead to a potential buffer overflow, but the results are truncated, and this is the source of the graphical display glitch, because (a) the prompt is truncated in the middle of a sequence of "invisible" characters, and (b) I think the readline code may have other bugs that cause character-counting not to work properly if the prompt itself is not completely present at the beginning of a buffer whose value is derived from the result of rl_message.

Solution:
rl_message ought to use a dynamically-allocated buffer instead, so it can adjust the size as needed. I'll attach a patch to provide this shortly.

Workaround:
The statically-allocated buffer is only used to store the final line in a multi-line prompt (including any invisible characters, and the special codes used by readline to mark the start and end of invisible-character sequences). Thus, if you add a newline in the prompt just before the "\$ ", the static buffer should have plenty of room. Similar methods might include not using 256 color support, or any other means to shorten the total size of the prompt string below 127.

I'll attach a patch that implements the described solution momentarily.

[Micah Cowan (micahcowan)]

This patch fixes the graphical display glitch.

In some instances, it seems that when I type "/", it still may not show up in some circumstances, so that would be a separate and remaining bug that still needs looking to, but the history search itself consistently and correctly shows the lines immediately after the prompt, which is all I really cared about.

The patch is quite straightforward, as it simply replaces the static buffer with a dynamically allocated one, and expands it when it needs to be expanded.

Problems will still arise on systems that don't support vsnprintf, as bash will quite happily overflow the buffer (static or dynamic), but there's little I can do about that, short of a more invasive rewrite; and readline is full of cases like that anyway. Anyway, no such situation will arise in Debian or Ubuntu, or any modern system for that matter.

[Micah Cowan (micahcowan)]

Er, there's a bug in that diff, in that it can use the same va_list twice, which isn't defined behavior. That might be why the "/" still doesn't show up. I'll fix that.

[Micah Cowan (micahcowan)]

Here's a patch that uses the double-vasnprintf call correctly. "/" still doesn't show up in all circumstances, but the code looks right to me, and does what it's supposed to.

[James Cuzella (trinitronx)]

I stumbled upon this bug when using a color PS4 prompt to trace bash scripts. Usually it would display fine, but I noticed some strange behavior once the shell level increased enough.

Bash repeats the first character of the PS4 prompt to indicate the shell level for scripts that call other scripts (or subshells). Once the shell level increases enough, the characters at the end of the prompt are truncated.

This causes display problems, especially if the "color off" ANSI code is chopped off the end.

Many thanks for finding this bug!
It was driving me nuts until I used the 'script' command to record my terminal's raw output and found that the prompt was being truncated.

With this patch, I still see that rl_message still seems to use a size of 128 and truncates if my system does not have vsnprintf. I compiled bash-static and tested a long PS4 with color codes, and it still seems to truncate for me.

How can I check that my system has vsnprintf?

[James Cuzella (trinitronx)]

Disregard my last question. I found this output in my build log:

checking for standard-conformant vsnprintf... yes

It appears that I must have not added my patch to the debian/patches folder the correct way. I didn't see that the patch was applied during the build log, and it was not listed in the stamps/ directory. So yeah, that was my bad ;-)

[Launchpad Janitor (janitor)]

This bug was fixed in the package readline6 - 6.2-7

---------------
readline6 (6.2-7) unstable; urgency=low

  * Fix build failure on hurd-i386 (Samuel Thibault). Closes: #644777.
  * Use dpkg-buildflags to set CFLAGS and LDFLAGS.

 -- Matthias Klose <email address hidden> Sun, 09 Oct 2011 11:23:45 +0200

[Matthias Klose (doko)]

should be fixed in 4.3, in 14.04 LTS