unwanted space after directory completion (probably old cruft after upgrades)

[latimerio]

With bash-completion 2.1, some users are seeing broken completion for directories, with a space being added at the end, requiring a [BACKSPACE]/[TAB].

Testcase copied from bug 1000470, which I'm closing because it was probably from acroread.sh, which is now blacklisted.

Extra space scenario:
find /op<TAB> ==> find /opt<Space>

Backslash scenario:
cd ~
mkdir -p test\ 1/test
find test\ 1/t<TAB>
find test 1/test
find: `test': No such file or directory
find: `1/test': No such file or directory

This bug can get closed when we find out what other old files get left in /etc/bash_completion.d and cause problems, so we can blacklist them, too.

[Peter Cordes (peter-cordes)]

This doesn't happen on a normal Trusty system, but yeah, there are similar reports of this happening to people. Must be some kind of leftover stuff from upgrades. My system has an upgrade chain going back to Edgy (6.10), but I keep it tidy with aptitude and other tools.

You may have some cruft in your /etc/bash_completion.d that's causing problems. Either from packages that were removed but not purged, or from obsolete conffiles (that the package owning them no longer supplies, but didn't remove during upgrade).

bug 1000470 looks like it was due to acroread's tab completion, which is blacklisted in Trusty. Does your bash_completion.d have a different name for acroread's adobe-supplied tab completion, that sneaks past the blacklist and breaks your programmable completion?

And check for obsolete conffile with:
dpkg-query -f '${Conffiles}\n' -W | grep 'obsolete$' | grep completion

I cooked up this perl script to go through the list and check if they match the stored md5sum from the package that provided them. And to print out the name of the package owning each one.
dpkg-query -f '${Package}\n${Conffiles}\n' -W |
perl -ane 'chomp; $pkg=$_ and next if !/^ /; $F[2] eq "obsolete" or next; -e $F[0] or print "no $F[0]\n" and next; ($localmd5)=split / /, `md5sum ${F[0]}`; print "$F[0] ". ($localmd5 eq $F[1] ? "un":"") . "modified in $pkg\n"; system "ls -l $F[0]";'

dpkg-query is just a fancy way to grep /var/lib/dpkg/status, but don't tell anyone, because it's Better to use the documented API (dpkg-query), instead of digging in the file yourself. The lines recording obsolete conffiles are removed by dpkg when the package owning them is purged, upgraded, or reinstalled. (not just reconfigured).

also see http://raphaelhertzog.com/2011/01/31/debian-cleanup-tip-1-get-rid-of-useless-configuration-files/ including one of the comments about obsolete conffiles leftover after package upgrades.

[Peter Cordes (peter-cordes)]

If we can figure out if there's one bad file that is commonly left lying around, and breaks things, a workaround can be shipped in bash-completion to blacklist it.

 I'm going to close any pre-trusty reports of this, since they're probably the acroread crap. And point those bugs to here.

I editted the title and description to make this bug be about what I think is probably the root cause. If you come across something that doesn't fit into what this bug is now, probably filing a new bug would be the best thing to do.

[Peter Cordes (peter-cordes)]

update: be careful with obsolete conffiles. The same file can be obsolete in one package, but still be provided and non-obsolete in another package. e.g. I removed my /etc/bash_completion.d/mercurial, since my script found it, but it had moved from mercurial to mercurial-common, and is listed non-obsolete by mercurial-common.

[Peter Cordes (peter-cordes)]

Attached the full-fledged perl script I cooked up to investigate obsolete conffiles.

grep the output, it doesn't take options.

When removing them, it often leaves empty directories, so don't forget to remove them, too. (If they aren't owned by some other package... check with dlocate, or add support to my script for walking up a path and checking.)

 I *highly* recommend etckeeper to save your bacon if you delete something and then wish you hadn't.
sudo git checkout master~4 -- bash_completion.d/mercurial
 was far more convenient than re-installing the package. And you have a record of what you did.

[latimerio (fomember)]

I have a freshly installed ubuntu 14.04.1 LTS and it also shows the problem.
Thus I doubt that it a legacy package.
I found that it occurs repeatable with symlinks and aliased commands as shown below

TEST SCENARIO
 ln -s /tmpX /tmp
 alias ll='ls -l'

ls /tmp<TAB> # gives "tmp/" and "tmpX/" without trailing spaces on a new line
ll /tmp<TAB> # ditto

ls /tmpX<TAB> # completes to "ls /tmpX/" on the same line

#BUT
ll /tmpX<TAB> # gives "ll /tmpX " with a trailing space thus breaking further completion

[latimerio (fomember)]

another observation as an addon to my previous post (I'd liked to edit it but unfortunately I can't)

less /tmpX<TAB> # gives "less /tmpX/" on the same line
type lv # gives "lv is /usr/bin/lv"
lv /tmpX<TAB> # gives "lv /tmpX " with a trailing space

So it is supposedly not the alias but some other mechanism.
Maybe it is a changed default for "unknown" commands.

P.S.
ls /etc/bash_completion.d # on my ubuntu 14.04.1 LTS gives
 apt-file axi-cache debconf gem1.9.1 grub initramfs-tools insserv pon ufw upstart

cat /etc/bash_completion # gives
. /usr/share/bash-completion/bash_completion

ll /usr/share/bash-completion/bash_completion # gives
-rw-r--r-- 1 root root 67659 Apr 7 2014 /usr/share/bash-completion/bash_completion

[Peter Cordes (peter-cordes)]

ln -s /tmpX /tmp produces:
lrwxrwxrwx 1 peter peter 5 Dec 9 11:58 /tmp/tmpX -> /tmpX (a broken symlink)

Did you mean ln -s /tmp /tmpX ?

When cooking up a testcase, it's bad practice to make one that only works for root.

mkdir -p test/tmp
cd test
ln -s ./tmp tmpX

ls ./tmp[TAB] => tmp/ tmpX/
echo ./tmp[TAB] => ditto

ls ./tmpX[TAB] => completes to ./tmpX/ (no space)
echo ./tmpX[TAB] => completes to ./tmpX/ (no space)

$ complete -p ls echo
complete -F _longopt ls
complete -F _minimal echo

so, works for me (even complete -r; . /.../pristine_ubuntu_trusty_bash_completion, since my normal bash-completion is git HEAD + my tweaks).

And btw, you're right that it's nothing to do with aliases. Anything without a specific completion has the shell function _minimal set for it by the completion dynamic loader. The _longopt function does end up using the same code to complete file / directory names as _minimal, it just checks for maybe completing --options first.

 Broken symlinks do complete like files (no slash, space added) instead of dirs (slash, no space). Since your ln command doesn't match your examples, maybe that's doing it.

ln -s /nowhere brokenlink
echo ./br[TAB] => ./brokenlink[space]
ls ./br[TAB] => ./brokenlink[space]

 I'm not sure how you're getting different behaviour from ls vs. a generic command. There is something weird going on here. Can you give ls -l output for the relevant stuff? If you can cook up a testcase that reproduces this in a tmp directory, that'd be better, otherwise just a full ls -l of all the relevant stuff on your system will do.

The other thing you can do is
set -x
ls /tmpX[TAB]
set +x

copy and paste the entire output. The interesting part will probably be where _filedir is called, and after that, but just post the whole output so we don't need you to understand it yourself. :P

[Peter Cordes (peter-cordes)]

Oh forgot to mention, you can toggle programmable completion on or off with
shopt -u progcomp # unset
shopt -s progcomp # set

Usually bash's builtin file/directory completion gets it right, and doesn't trip up on unusual characters in filenames (e.g. 'foo*' leads to problems with progcomp.) When progcomp does something weird, it's useful to see if bash-builtin completion does what you expect.

[latimerio (fomember)]

> Did you mean ln -s /tmp /tmpX ?

Sorry. Yes I meant ln -s /tmp /tmpX.
Thanks for pointing me to do tests as non root.
I usually don't work on the desktop but only do admin tasks over ssh.

We have a link /applics -> /nfs/AppServer/applics on all our clients.
If I do as regular user: ls /applics<TAB> or ll /applics<TAB> , I get a '/' appended as expected in both cases
If I do as root: ls /applics<TAB> I also get the '/' appended
  but if I do as root: ll /applics<TAB> I get a space appended

Our site has about 250 hosts with HP-UX, RH9, RHEL4, RHEL5, ubuntu 10, ubuntu 12 and ubuntu 14 (all in LTS) and the described problem only occurred with the arrival of ubuntu 14.

[Peter Cordes (peter-cordes)]

still can't reproduce. Is there any chance /nfs/AppServer/applics sometimes looks like a broken symlink? Do you get the same behaviour if you
ln -s /tmp /tmpX
and then do your examples exactly as typed? And ideally, can you do it without writing to /?
mkdir -p test/tmp; cd test
ln -s ./tmp tmpX
 so I don't have to sudo every time I want to create your testcase, or clean up my system. This is what I meant by a testcase that works for non-root.

 If this is only reproducible with absolute paths starting with /, then that might be a clue to something. I don't *think* mountpoints should come into this, but I can't reproduce this on my system. Does root on your system have anything different in /root/.bashrc? (or /root/.profile, if you're in a login shell as root).

 This space-appended behaviour is exactly what's supposed to happen for a broken symlink, so I still think something about your testcase is probably weird. You're not doing completions in a different context than the command will run, are you? like complete on one computer a command that you'll run on another with ssh?

 I know you probably didn't do anything silly like that, but I'm grasping at straws here.

Prob best if you can just capture some info to upload, from the shell that's behaving badly:

set > shell_defs.log # capture all your shell function definitions

set -x
ll /applics[TAB] # will spew hundreds of lines of shell code tracing
set +x
complete -p ls ll
ls -dl /applics /nfs/AppServer/applics

copy/paste that output to a file. (or do the above after running script. Yes, a program called script, it logs interactive sessions, default filename is typescript.)

 Err on the side of pasting more stuff, because so far this doesn't make sense.

[latimerio (fomember)]

Sorry for not getting back earlier.
I think I have found the cause for the problem at our site.

As I said we have a mixed environment with different versions of ubuntu and redhat.
In our common bashrc there is a check if the variable $BASH_COMPLETION is set.
On Redhat and until ubuntu 12 this variable gets set to /etc/bash_completion if bash_completion is sourced.
This is not the case with ubuntu 14 so our bashrc sources some legacy bash_completion from an NFS mount.

For ubuntu 14 this legacy stuff adds the trailing space in our case.

Thus I think this bug can be closed.

Nonetheless I'd like to know if there is way to detect if bash_completion is already sourced other than checking the BASH_COMPLETION variable.

[Peter Cordes (peter-cordes)]

> Sorry for not getting back earlier.

No worries. Glad you got your systems sorted.

Not closing this bug, because your problem wasn't this bug after all.. >.< I still want to leave this bug here for the purpose I editted the OP to.

> Nonetheless I'd like to know if there is way to detect if bash_completion is already sourced other than checking the BASH_COMPLETION variable.

Ubuntu 14.04 and git HEAD (of bash-completion) both set BASH_COMPLETION_COMPAT_DIR as a read-only shell variable.
The problematically-located /etc/profile.d/bash_completion.sh checks that. (problematic because bash_completion needs to be sourced for every shell, not just login shells.)

Until COMPAT_DIR support goes away (the /etc/bash_completion.d/ files that are sourced at startup, instead of dynamically when needed), this will be fairly foolproof, unless it was introduced too recently to be present on your old systems.

Fairly completely foolproof would be to check if there is a completion loaded for ls.

if ! complete -p ls &> /dev/null;
  source something
fi