non-ASCII characters in PS1 cause cursor to appear in wrong place

[Peter Berry]

Binary package hint: bash

Using the following prompt:

PS1='\[\e[0;32m\][\u@\h \W]\$ \[\e[m\]'

If the \W (i.e., working directory) contains a non-ASCII character, the cursor is placed 10 places too far to the right. If you then start typing, the cursor moves to the correct position (2 places right of the $) but leaves the first character where you typed it.

Reproduced in xterm, so is not specific to gnome-terminal; and not in python, so is (probably) not a bug in libreadline.

This is Feisty with all updates as of now.

[C de-Avillez (hggdh2)]

Thanks for taking the time to report this bug. Could you pleae provide an example for the bug?

[C de-Avillez (hggdh2)]

I researched further on this (and was helped by Nafallo and persia in #ubuntu-bugs).

Eventually I found this answer from the upstream maintainer, in the bash FAQ (ftp://ftp.cwru.edu/pub/bash/FAQ, question E3):

"E3) When I have terminal escape sequences in my prompt, why does bash
    wrap lines at the wrong column?

Readline, the line editing library that bash uses, does not know
that the terminal escape sequences do not take up space on the
screen. The redisplay code assumes, unless told otherwise, that
each character in the prompt is a `printable' character that
takes up one character position on the screen.

You can use the bash prompt expansion facility (see the PROMPTING
section in the manual page) to tell readline that sequences of
characters in the prompt strings take up no screen space.

Use the \[ escape to begin a sequence of non-printing characters,
and the \] escape to signal the end of such a sequence."

[C de-Avillez (hggdh2)]

So... for bug 110096, for example, the reporter was using the following prompting string:

PS1='work:\e[0;31m\w\e[m # '

Which falls straight into question E3, and gives out the repositioning issue.

By changing the PS1 string to enclose non-printing characters in '\[' and '\]' delimiters, like:

PS1='work:\[\e[0;31m\]\w\[\e[m\] # '

the issue is resolved.

[C de-Avillez (hggdh2)]

Rejecting: although the issue is obscure enough to warrant an entry in the bash FAQ, it is still not a bug.

[Peter Berry (pwberry)]

Quoting from the original report:

PS1='\[\e[0;32m\][\u@\h \W]\$ \[\e[m\]'

I have already put the \[ and \] sequences in.

[Peter Berry (pwberry)]

reopening since the reason for rejecting was invalid.

[C de-Avillez (hggdh2)]

Please provide an example for the bug.

[Peter Berry (pwberry)]

Example:

[pwb@rhuidean music]$ cd Motörhead/
[pwb@rhuidean Motörhead]$

The cursor is 11 places to the right of the $, i.e. 10 places too far right. Now type "echo foo":

[pwb@rhuidean Motörhead]$ cho foo e
foo

When you type e, it appears where the cursor is, then the cursor moves to 2 places right of the $.

I just found out that with a non-ASCII character in the PS1 like this, typing C-u while there is at between 1 and 9 characters on the command line, or backspace when there is only one, causes the cursor to jump another 10 places to the right.

[C de-Avillez (hggdh2)]

Ah well. I had added a comment here, and it seems it went to the land of the CPU ancestors.

I got it, sorry. I tried here with Derrière as the directory, and I was able to reproduce your bug. I then kept on hitting the up/down arrows to navigate thru the history, and the cursor kept moving more and more to the right. Rather interesting.

I will dig a bit further, and check upstream; right now, though, I guess we can set it as confirmed.

[C de-Avillez (hggdh2)]

Peter, please paste in here the output of your 'locale'. I think it may have a bearing on this behaviour.

Also, I cannot seem to be able to reproduce it with the following PS1 string *and* LANG=en_US.UTF-8:

PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\e[0;31m\]\w\[\e[m\] $ '

But, with your PS1 string and the same locale as above, it fails as stated... curiouser and curiouser...

[Peter Berry (pwberry)]

Locale is as follows:

LANG=en_GB.UTF-8
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=

[Mislav (mislav-marohnic+ubuntu)]

Mine is the same as above, except that it's en_US instead of GB. I'm able to reproduce it too with directories that contain Croatian chars.

[C de-Avillez (hggdh2)]

@Peter: I have just changed your PS1 string to

   PS1='\[\e[0;32m\][\u@\h \W]\$ \[\e[m\] '

Notice that the *only* difference is that I added a space at the end of the PS1 string, after the close escape. I cannot reproduce your issue anymore. Can you please test it?

@Mislav: I do not know if your PS1 string is similarly based -- if it is, can you please change as above? Otherwise, please copy it in here.

@All: if this works... we have a bypass, not a solution. The bug will still exist.

[Mislav (mislav-marohnic+ubuntu)]

I've used your PS1 string (both last and the one before). I still CAN reproduce, and for some reason it's worse then ever! Cursor is shifting wildly left-right when I type.

[C de-Avillez (hggdh2)]

Mislav, what is your version of bash and readline-common/libreadline?

[Mislav (mislav-marohnic+ubuntu)]

GNU bash, version 3.2.13(1)-release (i486-pc-linux-gnu)
readline-common/libreadline are 5.2

[Peter Berry (pwberry)]

hggdh, that PS1 does indeed work around the bug. Removing the space immediately after the $ (so the space appears outside the escapes) also works.

[Daniel Hahler (blueyed)]

I have backported bash from Gutsy (3.2-0ubuntu8; bash --version: 3.2.17(1)-release) and cannot reproduce it here:
[daniel@base ~]$ COMMAND_PROMPT=
[daniel@base ~]$ PS1='\[\e[0;32m\][\u@\h \W]\$ \[\e[m\]'
[daniel@base ~]$ mkdir Motörhead
[daniel@base ~]$ cd Motörhead/
[daniel@base Motörhead]$ echo foo
foo
[daniel@base Motörhead]$ echo $LANG
de_DE.UTF-8

This may have been fixed by http://ftp.gnu.org/gnu/bash/bash-3.2-patches/bash32-016 - which is in Gutsy already.

[C de-Avillez (hggdh2)]

Indeed I cannot reproduce it anymore on Gutsy's bash.

@Peter, Mislav: can you please check on this (if you are running Gutsy, or can backport bash)?

[Peter Berry (pwberry)]

I backported gutsy's bash using prevu, and the bug seems to have gone away.

[Daniel Hahler (blueyed)]

Fix appears to be released for Gutsy.