complete -o filename can't be used with filenames relative to another directory to handle space vs. trailing /

[hakon]

I am trying to ignore certain directory names when using bash completion. For example to ignore "backup" directories I use

$ cat setup.sh
_compTest() {
    local curdir cur words val name
    cur="${COMP_WORDS[$COMP_CWORD]}"
    baseFolder="."
    curdir=$PWD
    cd "$baseFolder"
    words=( $(compgen -f "$cur") )
    COMPREPLY=()
    for val in "${words[@]}" ; do
        name=$(basename "$val")
        if [[ $name == "backup" ]] ; then
            continue
        fi
        COMPREPLY+=( "$val" )
    done
    cd "$curdir"
}
complete -o filenames -F _compTest aaa

After sourcing this:

$ . setup.sh

I can type

$ aaa <tab><tab>

and it works fine. However if I use another baseFolder, for example setting

baseFolder="$HOME/base"

in the above script (setup.sh) (where $HOME/base is different from the current directory) the completion list is no longer what I would expect. That is:

* slashes are missing at the end of directory names, and
* a space is added after each completed directory name, instead of a slash ("/")

$ lsb_release -rd
Description: Ubuntu 14.04.1 LTS
Release: 14.04

$ apt-cache policy bash-completion
bash-completion:
  Installed: 1:2.1-4
  Candidate: 1:2.1-4
  Version table:
 *** 1:2.1-4 0
        500 http://no.archive.ubuntu.com/ubuntu/ trusty/main i386 Packages
        100 /var/lib/dpkg/status

[Peter Cordes (peter-cordes)]

This isn't bash-completion's bug. If anything, it's bash's bug, if you're sure that COMPREPLY is being mishandled.

More likely, it's a bug in your own script, since the usual programmable completion code jumps through some hoops do to the Right Thing for files and directories. Your script has huge problems with spaces in filenames, because they get quoted in compgen's output, and never unquoted before you pass them to COMPREPLY.

Incidentally, it would run WAY faster if you used a glob match instead of forking basename. See the hints in /usr/share/doc/bash-completion/README.gz

Also, pushd / popd would be easier than saving CWD. And safer still do do the whole thing inside a ( subshell ) to never affect the directory context of the user's shell. Even pushd/popd modify the user's $OLDPWD.

So you're trying to complete the current wod as a filename in a fixed directory, regardless of the current PWD, right?

The problem might be that you're using complete -o filenames. The results of your completion WON'T be filenames when your function returns, unless they're absolute paths. The space at the end instead of / is because you didn't use -o nospace, and bash didn't identify the word as a filename, because in the context of the shell when your function returns, stat("filename") gets ENOENT. bash only looks at COMPREPLY once you're done building it, not with the directory context you had while you were building it.

 If you backspace and put in a /, it will complete files in subdirectories of wherever you were.

 Anyway, as far as I can tell, everything is working as documented, your script is just using it wrong. :(

BTW, you can get the idental result to your script with MUCH less code and overhead from
_compTest() {
    local curdir cur words val name
    cur="${COMP_WORDS[$COMP_CWORD]}"
    baseFolder=~/"tmp"
    COMPREPLY=( $(cd "$baseFolder"; compgen -X '?(*/)backup' -f "$cur") )
}
complete -o filenames -F _compTest aaa

$( command substitution ) is already a subshell, so you can cd in there without affecting $OLWPWD or anything else in the user's shell.

 Of course, this still doesn't handle filenames with anything in their name, even spaces.

To do that, borrow code from bash-completion's _filedir function:

_compTest() {
    local IFS=$'\n'
    local cur="${COMP_WORDS[$COMP_CWORD]}"
    local quoted x

    baseFolder=~/"tmp"
    _quote_readline_by_ref "$cur" quoted
# COMPREPLY=( $(cd "$baseFolder"; compgen -X '?(*/)backup' -f "$cur") )

# IDK why _filedir does directories first.
    x=$( cd "$baseFolder"; compgen -d -X '?(*/)backup' -- "$quoted" ) &&
    while read -r tmp; do
        COMPREPLY+=( "$tmp" )
    done <<< "$x"

    x=$( cd "$baseFolder"; compgen -f -X '?(*/)backup' -- "$quoted" ) &&
    while read -r tmp; do
        COMPREPLY+=( "$tmp" )
    done <<< "$x"
}
complete -o filenames -F _compTest aaa

 Still doesn't know to put / after directory names, but does handle any filenames except those containing a newline. If you want the / for filenames thing, you'd have to write it yourself without depending on complete -o filenames to do it for you. I don't think that's possible for words that don't exist in the filesys...

[hakon (hakon-hagland)]

Thanks for the thorough answer! If I add "shopt -s extglob" your code is working.

The question is how to add slashes after directory names and not file names. If I remove "-o filenames" and add
"compopt -o nospace" I still can use "compgen -f "$cur" " but it does not add slashes. And the completions are not broken at "/" (if I insert it manually) such that I get a completion list with "a/b/c" instead of just "c" when I try to complete "a/b"..

Do I have to add a slash to $COMP_WORDBREAKS variable?

There is a post a stackoverflow :
http://stackoverflow.com/questions/10528695/how-to-reset-comp-wordbreaks-without-effecting-other-completion-script

It refers to a function "_get_comp_words_by_ref" .. where is this function documented? I searched the bash manual,
http://www.gnu.org/software/bash/manual/bashref.html
but it is not mentioned there..

Thanks!

[Peter Cordes (peter-cordes)]

Was working on completions for something else, and thought of this when I saw completions/make checking if COMPREPLY has only a single entry. If it does, that means completion will replace the user's word, rather than show possible completions. In that case, run compopt -o nospace.

 And maybe also make sure your possible completions for directories end with /, to trick the shell into appending that. (Sort that out in the subshell that's cded there, so it can tell.)

_get_comp_words_by_ref is in /usr/share/bash-completion/bash_completion. You might as well use any of those building blocks. But mostly they're overkill solutions for the problem of noticing when there's a > or < redirect, and completing on filenames for it, and then adjusting the rest of the line like it hadn't happened, I think.

 AFAICT, that's all you miss out on with
local cur="${COMP_WORDS[$COMP_CWORD]}"
vs.
 _get_comp_words_by_ref cur prev

If you want more help with this, you should prob. post on stackoverflow about how to complete filenames relative to something other than the current directory. This Ubuntu bug probably isn't the most visible place for anyone to notice and tip you off to a good idea. Since I'm pretty much at the limit of my knowledge of obscure tips and tricks for making something like this not take a lot of code and debugging time.

[hakon (hakon-hagland)]

Thanks Peter! See follow-up question at stackoverflow:

http://stackoverflow.com/questions/27244300/how-to-complete-filenames-relative-to-another-directory

[hakon (hakon-hagland)]

Hi Peter, you were completely right! Now I have got some help at stackoverflow, and it seems that helped solve this issue. There are four things that seems must be in place to solve it:

1. compopt -o nospace
2. compopt -o filenames
3. Running compgen -f with an absolute path
4. Then postprocessing the results of compgen
  a) removing absoulte path prefix
  b) adding slashes after filenames

So here is minimal example (not handling quoting and spaces in file names):

_compTest() {
    local cur tmp val
    cur="${COMP_WORDS[$COMP_CWORD]}"
    tmp=~/tmp/
    tmp_escaped=${tmp//\//\\\/}
    cur="$tmp$cur"
    COMPREPLY=( $(compgen -f "$cur") )
    for ((i=0 ; i<${#COMPREPLY[@]}; i++ )) ; do
        val="${COMPREPLY[$i]}"
        [[ -d "$val" ]] && val="$val/"
        COMPREPLY[$i]="${val/$tmp_escaped/}"
    done
}
complete -o filenames -o nospace -F _compTest aaa

Thanks again for excellent help!