Ubuntu

bash completion now quotes shell variable references rather than expanding them

Reported by Dan Krejsa on 2011-05-06
428
This bug affects 87 people
Affects Status Importance Assigned to Milestone
bash (Debian)
Fix Released
Unknown
bash (Fedora)
New
Undecided
Unassigned
bash (Ubuntu)
Undecided
Unassigned
Precise
Undecided
Unassigned
Quantal
Undecided
Unassigned
Raring
Undecided
Unassigned

Bug Description

Binary package hint: bash

NB! NB! NB!

After applying the update to resolve this problem one _also_ must enable direxpand option by issuing:

shopt -s direxpand

If you do not do this, the variable expansion will not happen.

NB! NB! NB!

[Impact]
Bash-4.2 tries to leave completed directory names as the user typed them,
without expanding them to a full pathname. One effect of this is that
shell variables used in pathnames being completed (e.g., $HOME) are left
unchanged, but the `$' is quoted by readline because it is a special
character to the shell.

[Test Case]
- [Current test case showing expected behavior]

Items to address so far:
  - that $ is escaped less
  - that direxpand does not expand as expected
  - that direxpand option is available and if enabled, expands directories with variable names.

zef:~/Projects/precise-amd64$ mkdir -p /tmp/testd/finger/toe
zef:~/Projects/precise-amd64$ cd /tmp/testd/
zef:/tmp/testd$ td=`pwd`
zef:/tmp/testd$ cd ../
zef:/tmp$ ls \$td/finger

zef:/tmp$ shopt -s direxpand
zef:/tmp$ ls $td/fing<TAB>
zef:/tmp$ ls /tmp/testd/finger/
toe

- [Previous Test Case showing non-working bash installation]
~$ mkdir -p /tmp/testd/finger/toe
/tmp$ cd /tmp/testd
/tmp/testd$ td=`pwd`
/tmp/testd$ cd ../
/tmp$ ls $td/fing

Type a TAB character at this point, and bash completion makes the line:
/tmp$ ls \$td/finger

Note that while 'fing' has been expanded to 'finger ' (with a space at the end), the $td variable reference has had the
$ character quoted with a backslash. This prevents further extension of the pathname, and in fact makes
the command invalid; if I type a newline at this point, I get:
ls: cannot access $td/finger: No such file or directory

This is not useful behavior. I believe that in Ubuntu 10.10, the $td variable would have been expanded
to '/tmp/testd', and subsequent completions starting from '/tmp/testd/finger' would have worked.

[The version of /etc/bash_completion.d/acroread.sh that is installed starts with:

# _filedir : to handle file and directories with spaces in their names.
if ! type _filedir &> /dev/null ; then

so I don't think that file is overriding /etc/bash_completion's _filedir ...]

[Regression Potential]
- (#32) low potential for regressions since this bug is related to the interactive command line and would not influence any scripts.

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: bash 4.2-0ubuntu3
ProcVersionSignature: Ubuntu 2.6.38-8.42-generic 2.6.38.2
Uname: Linux 2.6.38-8-generic x86_64
Architecture: amd64
Date: Fri May 6 09:48:29 2011
InstallationMedia: Ubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
ProcEnviron:
 SHELL=/bin/bash
 PATH=(custom, user)
 LANG=en_US.UTF-8
 LANGUAGE=en_US:en
SourcePackage: bash
UpgradeStatus: Upgraded to natty on 2011-04-30 (6 days ago)

Dan Krejsa (dan-krejsa) wrote :
Norbert (nrbrtx) wrote :

Fixed by editing line 1587 of /etc/bash_completion (see https://bugs.launchpad.net/ubuntu/+source/bash/+bug/769866/comments/10)

Norbert (nrbrtx) wrote :

My comment #2 was incorrect, sorry.

Trailing space and quoting backslash have not disappeared after doing #2.

This bug is really ugly. I often use 'll $HEAD/' <tab><tab> which now doesn't work anymore. Unfortunately I don't hace a 10.10 system anymore to compare...

Richard Lee (rdlee632) wrote :

This bug is really horrible. Anyone know what line in bash_completion I need to fix? How did it make it into the release??? Also, I sometimes see completion adding an extra space when completing directory names, which makes it difficult to traverse through a directory hierarchy with tab completion.

Changed in bash (Ubuntu):
status: New → Confirmed
vjrj (vjrj) wrote :

As a workaround while this is fixed I've downgraded to maverick package:
http://packages.ubuntu.com/maverick/bash

Solganik (solganik) wrote :

same here .. had to downgrade to maverick package .. hope it will be fixed.

Since a couple of weeks on 10.11 and still the same behavior. Who maintains bash?

Rudy Vissers (rudy-vissers) wrote :

Please... Using the tab is much important for the developers/administrators. We can't remove the backslash every time we use the tab ! It is a bug that can't be in a supported version ! (12.0.4 LTS). It is the kind of things that makes you hate an environment ! Because we use the tab ALL THE TIME !

mumeiyamibito (mumeiyamibito) wrote :

Who makes such specification?
And who come out ahead this specification (or bug) ?

If someone know reason that this specification is better than before one, please tell me.

toadwarble (jmc-xisl) wrote :

This is a real pain for me as I set 101 variables to point to directories I want to get at and am in the habit of typing $foodir/abc<tab> to get what I want and now I can't do it so well.

nacorn (acorn) wrote :

SOLUTION: This was a bug in bash which is fixed in patch bash4.2-029. See
http://lists.gnu.org/archive/html/bug-bash/2012-07/msg00018.html

I build from source as follows:
  - downlaod bash-4.2.tar.gz from ftp://ftp.gnu.org/pub/gnu/bash/
  - extract
  - download patches bash42-001 through bash42-029 (actually I did all through 037) from ftp://ftp.gnu.org/pub/gnu/bash/bash-4.2-patches/
 - apply patches
 - ./configure --prefix=/usr/local; make; make install

Using this /usr/local/bash fixes the problem -- tab completion on environment variables works again.

Can these patches please be applied to the package in precise and oneiric?
Thanks!

nacorn (acorn) wrote :

Note: This bug affects 12.04 Precise (not just natty). The solution in comment 14 was tested on 12.04

Carter Sanders (gamma3) wrote :

I upgraded to bash 4.2.36(1), and the problem is only half fixed. On older versions of bash (certainly in the bash2 vintage), cd $VARIABLE/<tab> offered completions. At lest in the newer version it doesn't escape the dollar sign anymore, but this is still a removed, highly used feature.

Bill Hammond (gellmu) wrote :

This bug was first reported in May 2011 and still not fixed?
It's driving me crazy. A long-established very helpful bash feature
has been trashed.

What's going on?

This is bad.

Bill Hammond (gellmu) wrote :

But, alas, it seems to be upstream in bash itself.

Axel Beckert (xtaran) wrote :

Interestingly I can reproduce this issue only on some machines, but not on all of them despite they all have bash 4.2.

I could reproduce it on Natty and Precise, on Debian Wheezy, and on one, but not on all of my Debian Sid boxes (4.2-5).

Disabling the sourcing of /etc/bash_completion (either manually enabled in /etc/bash.bashrc or ~/.bashrc) made the problem vanish on Debian and on Natty, but not on Precise. Even purging bash-completion did not help on Precise.

And that's not all: While on precise using "bash --rcfile /dev/null" still did not fix the issue, it brought the issue back on Natty. And on Debian Sid it still worked as expected, i.e. did not add any backslash.

Changed in bash (Debian):
status: Unknown → New
KSSG (kssg) wrote :

I am also suffering this problem in Ubuntu 12.10.
Commenting out the sourcing of bash-completion managed to fix it, but the problem is that it disables all of its advantages. The problem must be in one of the various files at /usr/share/bash-completion/completions ,I'd bet. But debugging that until finding the culprit would be a task of true patience.

I tried to build bash manually and it always does the same thing unless I disable sourcing bash-completion.

Alec Warner (antarus) wrote :

Ok:

This bug does not affect Q or R (I checked.)

I make an internal package (for myself), and it resolves the problem.
I've attached a debdiff which creates bash-4.2-2ubuntu2-gg1; it should be trivial to modify it to produce bash4.2-2ubuntu3.

Thomas Ward (teward) wrote :

Nominated for Precise. As Comment 21 says Quantal and Raring are unaffected, i'm not nominating them.

Given that this bug was initially filed against Natty, can someone test this on Oneiric to see if it is also affected?

------
Thomas Ward
Ubuntu Bug Squad Member

Alec Warner (antarus) wrote :

antarus@antarus-glaptop:~$ cd Downloads/
antarus@antarus-glaptop:~/Downloads$ ls
bash_4.2-0ubuntu4_amd64.deb
bash_4.2-2ubuntu2-gg1.debdiff.txt
bash_4.2-5ubuntu2_amd64.deb
antarus@antarus-glaptop:~/Downloads$ ar x bash_4.2-0ubuntu4_amd64.deb
antarus@antarus-glaptop:~/Downloads$ ls
bash_4.2-0ubuntu4_amd64.deb control.tar.gz
bash_4.2-2ubuntu2-gg1.debdiff.txt data.tar.gz
bash_4.2-5ubuntu2_amd64.deb debian-binary
antarus@antarus-glaptop:~/Downloads$ tar -zxf data.tar.gz
antarus@antarus-glaptop:~/Downloads$ ls
bash_4.2-0ubuntu4_amd64.deb control.tar.gz etc typescript.txt
bash_4.2-2ubuntu2-gg1.debdiff.txt data.tar.gz usr
bash_4.2-5ubuntu2_amd64.deb debian-binary1_i386.deb
bin
antarus@antarus-glaptop:~/Downloads$ bin/bash
eantarus@antarus-glaptop:~/Downloads$ echo \$HOME

Oneric also appears affected by this bug.

Graeme Hewson (ghewson) wrote :

I'm on Quantal (Kubuntu) and I'm affected by the bug. See also comment 20.

Alec Warner (antarus) wrote :

I need more details from Quantal users then. Quantals' bash has bash42-029.diff, which fixes the bug in bash that caused a (and I presume this) problem. It could be Quantal has another bug?

I just downloaded bash_4.2-5ubuntu1_amd64.deb and could not reproduce. What version of bash are you using?

-A

Graeme Hewson (ghewson) wrote :

$ bash --version
GNU bash, version 4.2.37(1)-release (x86_64-pc-linux-gnu)

$ wajig list bash
ii bash 4.2-5ubuntu1 amd64 GNU Bourne Again SHell
ii bash-completion 1:2.0-1ubuntu2 all programmable completion for the bash shell
ii command-not-found 0.3ubuntu5 all Suggest installation of packages in interactive bash sessions

$ wajig detail bash
Package: bash
Essential: yes
New: yes
State: installed
Automatically installed: no
Multi-Arch: foreign
Version: 4.2-5ubuntu1
Priority: required
Section: shells
Maintainer: Ubuntu Developers <email address hidden>
Architecture: amd64
Uncompressed Size: 1,450 k
Depends: base-files (>= 2.1.12), debianutils (>= 2.15)
PreDepends: dash (>= 0.5.5.1-2.2), libc6 (>= 2.15), libtinfo5
Recommends: bash-completion (>= 20060301-0)
Suggests: bash-doc
Conflicts: bash-completion (< 20060301-0), bash-completion (< 20060301-0), bash
Replaces: bash-completion (< 20060301-0), bash-completion (< 20060301-0), bash-doc (<= 2.05-1), bash-doc (<= 2.05-1)
Provides: bash
Provided by: bash
Description: GNU Bourne Again SHell

$ wajig detail bash-completion
Package: bash-completion
State: installed
Automatically installed: no
Multi-Arch: foreign
Version: 1:2.0-1ubuntu2
Priority: standard
Section: shells
Maintainer: Ubuntu Developers <email address hidden>
Architecture: all
Uncompressed Size: 892 k
Depends: bash (>= 3.2)
PreDepends: dpkg (>= 1.15.7.2~)
Breaks: xen-tools (<= 4.1-1), xen-tools (<= 4.1-1)
Replaces: bash, bash, cryptsetup (< 2:1.1.2-2), cryptsetup (< 2:1.1.2-2), xen-tools (<= 4.1-1), xen-tools (<= 4.1-1)
Provides: bash-completion
Description: programmable completion for the bash shell

Dimitri John Ledkov (xnox) wrote :

This bug was fixed in 4.2-3 upload, and hence is fixed in quantal and raring.

However, there is a new option that one needs to set - direxpand.

direxpand
 If set, bash replaces directory names with the results of word expansion when performing filename completion. This
 changes the contents of the readline editing buffer. If not set, bash attempts to preserve what the user typed.

bash (4.2-3) unstable; urgency=low

  * Apply upstream patches 025 - 029, fixes for
    - When used in a shell function, `declare -g -a array=(compound assignment)'
      creates a local variable instead of a global one.
    - The `lastpipe' option does not behave correctly on machines where the
      open file limit is less than 256.
    - When the `extglob' shell option is enabled, pattern substitution does not
      work correctly in the presence of multibyte characters.
    - When using a word expansion for which the right hand side is evaluated,
      certain expansions of quoted null strings include spurious ^? characters.
    - Bash-4.2 tries to leave completed directory names as the user typed them,
      without expanding them to a full pathname. One effect of this is that
      shell variables used in pathnames being completed (e.g., $HOME) are left
      unchanged, but the `$' is quoted by readline because it is a special
      character to the shell. This patch introduces two things:
      - A new shell option, `direxpand', which, if set, attempts to emulate the
        bash-4.1 behavior of expanding words to full pathnames during
        completion;
      - A set of heuristics that reduce the number of times special characters
        such as `$' are quoted when the directory name is not expanded.
  * Mark bash and bash-static as Multi-Arch: foreign . Closes: #674120.
  * Avoid warning for unset debian_chroot with -u (Marko Mäkelä).
    Closes: #659940.
  * Fix string expansion: \u and \U for code points U+0080 to U+00FF (Darran
    Salt). Closes: #661386.
  * Search $- for `i' for the interactivity test. Closes: #482194.
  * Search for bash_completion in /usr/share/bash-completion too.
    Closes: #655071, #655208.

 -- Matthias Klose <email address hidden> Sun, 01 Jul 2012 13:45:35 +0200

Changed in bash (Ubuntu Raring):
status: Confirmed → Fix Released
Changed in bash (Ubuntu Quantal):
status: New → Fix Released
Changed in bash (Ubuntu Precise):
status: New → Incomplete
Dimitri John Ledkov (xnox) wrote :

I do not believe the fix is a valid SRU candidate, because it changes quoting behaviour, introduces a new option and requires a user to opt-in / change that option.

Dimitri John Ledkov (xnox) wrote :

If you need a fix for the bug in precise, please update the bug description as per Stable Release Updates template.

[1] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

Dimitri John Ledkov (xnox) wrote :

@Alec Warner (antarus):
The proposed patch is ok, sans missing this bug reference and not using correct SRU version number. But that's easy enough to fix when sponsoring. Please update bug description as per SRU guidelines if you want this uploaded in Precise. Thanks.

Dimitri John Ledkov (xnox) wrote :

unsubscribing sponsors, please resubscribe when the bug description is updated as per SRU guidelines.

JThoennes (joerg-thoennes) wrote :

[Impact]
 * Explanation see above.
 * This is really a bug, ie the behaviour expected by the user changes unexpectedly.
 * In addition, only the interactive command line is affected, bash scripts should not see this change.
[Test Case]
 * See above.
[Regression Potential]
 * I see a low potential for regressions since this bug is related to the interactive command line and would not influence any scripts.
[Other Info]

So IMHO this is a very good candidate for Precise.

Dimitri John Ledkov (xnox) wrote :

@ joerg:
but the default interactive behaviour has changed
$ shopt direxpand
direxpand off

And one needs to manually re-enable that option to get previous behaviour.

The template you posted lacks required information. Specifically:

The test case should check:
- that $ is escaped less
- that direxpand does not expand as expected
- that direxpand option is available and if enabled, expands directories with variable names.

The impact should describe what is being changed (underquoting + new option).

Regression potential w.r.t. to underquoting should be considered (e.g. security issues?!)

Changed in bash (Debian):
status: New → Fix Released
Adam Stokes (adam-stokes) wrote :

If anyone still has a reproducer environment setup with a patch bash version on precise and can easily reproduce please feel free to do so and alter the description above. In the meantime, Ill get a reproducer setup and update the description once my test is done.

Thanks
Adam

description: updated
Changed in bash (Ubuntu Precise):
status: Incomplete → In Progress
description: updated
description: updated
Adam Stokes (adam-stokes) wrote :

Updated debdiff to include bug lp in changelog

Dimitri John Ledkov (xnox) wrote :

$ dput ubuntu bash_4.2-2ubuntu2.1_source.changes
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading bash_4.2-2ubuntu2.1.dsc: done.
  Uploading bash_4.2-2ubuntu2.1.diff.gz: done.
  Uploading bash_4.2-2ubuntu2.1_source.changes: done.
Successfully uploaded packages.

The package was uploaded into precise-proposed unapproved queue, pending for a member of the SRU team to accept the package into the precise-proposed archive.

Hello Dan, or anyone else affected,

Accepted bash into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/bash/4.2-2ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in bash (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed

Hi,

I installed bash 4.2-2ubuntu2.1 from precise-proposed.
Enabling the direxpand bash shell option fixes this issue to my satisfaction, thanks!

- Dan

tags: added: verification-done
removed: verification-needed
Daniel Drucker (dmd-c) wrote :

Yes, this fixes it.

summary: - In natty, bash completion now quotes shell variable references rather
- than expanding them
+ bash completion now quotes shell variable references rather than
+ expanding them

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package bash - 4.2-2ubuntu2.1

---------------
bash (4.2-2ubuntu2.1) precise-proposed; urgency=low

  * Fix directory expansion. (LP: #778627)
 -- Alec Warner <email address hidden> Mon, 18 Mar 2013 22:12:54 +0000

Changed in bash (Ubuntu Precise):
status: Fix Committed → Fix Released
Markus Klyver (markusklyver) wrote :

Finally, this bug is fixed!

Ladios Jonquil (ladios) wrote :

Any idea why `ls $d/f[TAB]` still becomes `ls \$d/f `? Other commands seem fine, but not `ls`.
Does anyone also have the same problem after update? I'm running Precise, btw

JThoennes (joerg-thoennes) wrote :

Ladios,

just updated my precise installation to the latest bash package and now it works as expected.
Including "ls".

Ladios Jonquil (ladios) wrote :

Investigated. All the commands that failed to do completion properly are defined with `-F _longopt`.
dpkg-query shows me that there are a lot of conffiles... OMG I wonder how many they are throughout my installation.
Will try to get rid of those conffiles to see if that fixes my problem.

Ladios Jonquil (ladios) wrote :

Oh... I thought conffiles is short for conflict files... lol

Ladios Jonquil (ladios) wrote :

Actually, most defined completions would add '\' in front of '$'. It only works when the command has no completion defined, for instance, the `ll` alias does.
Don't know what went wrong. I think I have to go for the direxpand option.

On 10 April 2013 14:48, Ladios Jonquil <email address hidden> wrote:
> Actually, most defined completions would add '\' in front of '$'. It only works when the command has no completion defined, for instance, the `ll` alias does.
> Don't know what went wrong. I think I have to go for the direxpand option.
>

Did you enable direxpand option?

$ shopt -s direxpand

All this SRU does is backport "direxpand" option and leaves it
disabled by default. One must enable it, if variable expansion is
desired.

Regards,

Dmitrijs.

description: updated
Ladios Jonquil (ladios) wrote :

Yes, I enabled direxpand option eventually. It works.
I misunderstood this update. I thought this would actually fix the shell variables quoting issue (by not quoting them), but it didn't.

Jeff Ward (jeff-ward) wrote :

I'd just like to chime in and say, whoever had the bright idea of escaping $ during tab completion... go to hell.

thava (thavamuni) wrote :

If the proposed fix requires setting direxpand variable, then that is not a fix, it is a workaround fix. (something is better than nothing though). Historical behaviour should not be changed and users must not be expected to set this new variable. This is affecting so many people and this is real pity.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.