bash utf-8 characters issue in xterm-like virtual terminals

[®om]

Bash/readline handles line editing incorrectly when multibyte characters appear in $PS1 between the "\e]0;" and "\a" escape sequences (which serve to update the title of the virtual terminal window).

For example, I created a folder "~/Vidéos/ééé".

the char 'é' takes 2 bytes in UTF-8.

When I write something, no problem :
rom@rom-laptop:~/Vidéos/ééé$ echo abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrst
uvwxyz

The problem appears when pressing ↑ (up) in the terminal to write again the same command, and editing it : everything is shifted of 4 chars, cursor is not at the right place, and what appears in the command is not the command executed (see attached screenshot).

[®om (rom1v)]

reported upstream : http://bugzilla.gnome.org/show_bug.cgi?id=580297

[®om (rom1v)]

This bug is very disturbing, for example :

rom@rom-desktop:~/Téléchargements/aMule$ echo abcdef
abc def
rom@rom-desktop:~/Téléchargements/aMule$ (then press ↑ (up), put the cursor on 'c' for example, and type something)

(It is important to have a directory with 2-bytes chars : Téléchargements)

[Matthias Klose (doko)]

please recheck with bash 4.0 in karmic.

[ZelinskiyIS (ivze)]

Greetings, everyone!

I am experiencing the same problem when using Russian letters (such as Абвгд... :)) in bash.
I am the creator of a separate bug report, quite an old one, which is about the same problem. I have linked the report to this one by marking my bug as a duplicate of this.

I have performed some investigantion into the problem. Here it comes, copypasted from bug #305554 :
#Copypaste#############################################################
I have performed some research about the unwanted editing behaviour and found out that the bug is localised in "bash" interpreter (GNU bash, version 3.2.48(1)-release).

I found that in case of bash, such behaviour is triggered by using Multibyte UTF-8 symbols in PS1 variable in a special way. Default ubuntu (tested for 9.04) .bashrc script works so that when bash is launched from within gnome-terminal, xterm or some other software terminals (bash looks at TERM variable), PS1 is set to \[\e]0;\u@\h: \w\a\]${debian_chroot:+($debian_chroot)}\u@\h:\w\$. The long construction includes a sequence \e]0;some text\a. The sequence is interpreted by software terminal as a command to change the window title to "some text". In a situation when current working directory has multibyte symbols in path, some multibyte characters happen to appear between \e]0; and \a. I performed some tests by means of manually setting PS1 values and found out that messy command line edting is triggered by appearance of any multibyte characters inside \e]0 and \a. Any characters outside those do not trigger the messy behaviour.

I have attached data, captured by "script" utility, that shows the bug. The utility reproduces data, that appeared on my terminal with timing preserved. To play the data, use a terminal that supports title changing (gnome-terminal and xterm must work), unpack the archive, ensure that terminal size is 80x24 and run "scriptreplay times typescript 2". In particular, you will see that having three two-byte unicode symbols betwee \e]0; and \a causes a three-position miss when editing a long line with the use of backspace. The test shows only a part of messy editing behavior. Some tests, described above in this bug report, have not been performed.

I suppose that this bug is in "readline" library. However, because readline is statically compiled into bash, I think that it should be treated as a bash bug.
#End of copypaste#############################################################

[ZelinskiyIS (ivze)]

The long text above is for Jaunty 9.04. I have rechecked the bug in Karmic beta for bash 4.0.33, and the bug is still here.

My steps to reproduce were:
1)Cd to a directory with multibyte characters in name
2)Write some command longer than terminal horizontal size.
3)Try to edit the line in different ways (move backwards by means of |<-|) and see messy behaviour. Press |/|\| to see the previous command, you typed, and see that the output is messy again.
4)Execute PS1=\$ and see that the bugs are no more with the different prompt.

[ZelinskiyIS (ivze)]

I have made a "script" record of reproducing this bug for Karmic beta. In the record you will see steps 1),2),3) reproduced. To play: 1)unpack; 2)cd to unpacked; 3) run "scriptreplay times" using a 80x24 terminal that supports title changing.

[Yann Leprince (sciyann)]

I have updated the description of this bug to reflect the results of the experiments performed by ZelinskiyIS (many thanks to him). According to the gnome-terminal people this is not a bug of the virtual terminal but likely a bug in bash/readline (see http://bugzilla.gnome.org/show_bug.cgi?id=580297 ). As the bash source contains its own copy of the readline code I think that the right people to contact are bash upstream developers.

[Gaël Eveno (gael-eveno)]

I've got the same problem with my fresh Linux From Scratch installation!
The only thing I can add on this problem is it only appears when I use a virtual terminal such as xterm and gnome-terminal (that use pts*). With a "real" terminal (tty*) the behaviour is as usual and works fine! As I use the same bash/readline in the different terminals with the same settings (i.e. everything with UTF-8, locales, etc...) I just wonder if the problem can't come from one of the Xserver's libraries. Changing the keyboard driver with Xorg doesn't resolve the problem (evdev/xkb etc...). Keys work fine with Gnome Desktop but something is wrong between the Xserver/Window manager and virtual terminals.

[Gaël Eveno (gael-eveno)]

I forgot to mention that I'm using bash 4.0 compiled with the already installed readline library (i.e. without the readline built-in). My readline library: readline-6.0

[Gaël Eveno (gael-eveno)]

Please forget my previous messages! It seems that my problem is not "exactly" the same than the bug described here! The strange behaviour with my key UP and DOWN let me believe it but I think my problem is a keymap problem as now I've got the problem with all gnome applications and not just with gnome-terminal and xterm as before. So forget the messages as it can be applied only for people having my problem!