bash-completion tilde expansion every time

[Ove Risberg]

[Impact]

when authenticate AD, with bask-completion, it's very slow.

[Testcase]

Below command should output nothing.

root@ubuntu1404:~# strace bash -c ". /etc/bash_completion; quote_readline '/something'; echo" 2>&1 | grep /etc/passwd
root@ubuntu1404:~# strace bash -c ". /etc/bash_completion; quote_readline '/something'; echo" 2>&1 | grep sss

but we have something

root@ubuntu1404:~# strace bash -c ". /etc/bash_completion; quote_readline '/something'; echo" 2>&1 | grep /etc/passwd
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
root@ubuntu1404:~# strace bash -c ". /etc/bash_completion; quote_readline '/something'; echo" 2>&1 | grep sss
open("/lib/i386-linux-gnu/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/var/lib/sss/mc/passwd", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
connect(4, {sa_family=AF_LOCAL, sun_path="/var/lib/sss/pipes/nss"}, 110) = 0

[Regeression Potential]

Regression possible about https://bugs.launchpad.net/ubuntu/+source/bash/+bug/1288314

but there is no problem with ~ completion in my test

I thested ~/<Tab>/ ~/<Tab>/<Tab> with no problem

[Other Info]

This is not from upstream, no need to fix for upstream.
But fix code seems harmless. just specify special character ~

// The original description

1) The release of Ubuntu you are using, via 'lsb_release -rd' or System -> About Ubuntu
root@ubuntu1404:~# lsb_release -rd
Description: Ubuntu 14.04.1 LTS
Release: 14.04

2) The version of the package you are using, via 'apt-cache policy pkgname' or by checking in Software Center
root@ubuntu1404:~# apt-cache policy bash-completion
bash-completion:
  Installed: 1:2.1-4
  Candidate: 1:2.1-4
  Version table:
 *** 1:2.1-4 0
        500 http://se.archive.ubuntu.com/ubuntu/ trusty/main i386 Packages
        100 /var/lib/dpkg/status

3) What you expected to happen

When I use bash-completion it should not read the /etc/passwd file or contact the Active directory.
It is very hard to see this problem unless you run this on a machine with slow authentication... but the commands below will show if the /etc/passwd or sssd are used by bash_completion or not:

root@ubuntu1404:~# strace bash -c ". /etc/bash_completion; quote_readline '/something'; echo" 2>&1 | grep /etc/passwd
root@ubuntu1404:~# strace bash -c ". /etc/bash_completion; quote_readline '/something'; echo" 2>&1 | grep sss

When the ~ are escaped on line 545 and 547 in /usr/share/bash-completion/bash_completion then bash_completion works as expected.

4) What happened instead

Every time you use tab-completion in bash on ubuntu 14.04 it will read the /etc/passwd file and if sssd are installed and configured for Active Directory authentication it will talk to sssd... and this can be slow in some environments.

root@ubuntu1404:~# strace bash -c ". /etc/bash_completion; quote_readline '/something'; echo" 2>&1 | grep /etc/passwd
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
root@ubuntu1404:~# strace bash -c ". /etc/bash_completion; quote_readline '/something'; echo" 2>&1 | grep sss
open("/lib/i386-linux-gnu/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/var/lib/sss/mc/passwd", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
connect(4, {sa_family=AF_LOCAL, sun_path="/var/lib/sss/pipes/nss"}, 110) = 0

I have made a small patch for this problem and I have not found any side effects:
--- /usr/share/bash-completion/bash_completion.orig 2014-11-06 10:36:14.981888369 +0100
+++ /usr/share/bash-completion/bash_completion 2014-11-06 10:36:25.142070963 +0100
@@ -542,9 +542,9 @@
     elif [[ $1 == \'* ]]; then
         # Leave out first character
         printf -v $2 %s "${1:1}"
- elif [[ $1 == ~* ]]; then
+ elif [[ $1 == \~* ]]; then
         # avoid escaping first ~
- printf -v $2 ~%q "${1:1}"
+ printf -v $2 \~%q "${1:1}"
     else
         printf -v $2 %q "$1"
     fi

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "bash_completion.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Ove Risberg (ove-risberg)]

Ubuntu 14.10 have the same problem:

root@ubuntu1410:~# strace bash -c ". /etc/bash_completion; quote_readline '/something'; echo" 2>&1 | grep /etc/passwd
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3

root@ubuntu1410:~# strace bash -c ". /etc/bash_completion; quote_readline '/something'; echo" 2>&1 | grep sss
open("/lib/x86_64-linux-gnu/libnss_sss.so.2", O_RDONLY|O_CLOEXEC) = 3
open("/var/lib/sss/mc/passwd", O_RDONLY|O_CLOEXEC) = 3
connect(4, {sa_family=AF_LOCAL, sun_path="/var/lib/sss/pipes/nss"}, 110) = 0

The /usr/share/bash-completion/bash_completion file are identical on 14.04 and 14.10 so the patch installs without any problem and the workaround works as expected.

[Peter Cordes (peter-cordes)]

This is fixed upstream. They reworked the relevant code, and the line that gratuitously used an unquoted ~ where it shouldn't have is gone.

[Ove Risberg (ove-risberg)]

What can we do to get this fixed for Ubuntu 14.04LTS?

We have a patch, it is fixed upstream, it is a really simple fix and it is really annoying if you use slow Active Directory authentication.

[David Hesselbom (david-hesselbom)]

I applied this fix some time ago and have not experienced the issue since, but today I installed some update (not sure what) and the issue re-appeared. I looked at /usr/share/bash-completion/bash_completion and sure enough, it was back the way it was before the fix. Apparently someone is making updates without including this fix...

[Keith Hanlan (keith-hanlan)]

David, from Peter's comment (#5), it seems that the changes to _quote_readline_by_ref() in bash_completion are no longer required. Essentially, the fix lies elsewhere. Have I got that right Peter?

[Ove Risberg (ove-risberg)]

I checked my Ubuntu 14.04 machine and the problem is not solved yet.

Run this command:
strace bash -c ". /etc/bash_completion; quote_readline '/something'; echo" 2>&1 | grep /etc/passwd

When the problem is solved you should get no output and now you get this output:
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 3

The output indicates that it tries to expand the ~ and this is not what was intended and if some kind of network authentication is used it will cause long delays... sometimes more then 10 seconds... so this is a serious problem.

Please apply the patch I provided 7 months ago when I reported this problem or test the fix they did upstream.

[Ove Risberg (ove-risberg)]

Now over one year after this problem was reported and a patch was provided the ticket is still not assigned to anyone.

This problem is a big pain for Enterprise users and it makes Ubuntu very slow when the terminal is used.

Please apply the patch or explain what we can do to get some attention to this problem

[Ove Risberg (ove-risberg)]

The problem is available on Ubuntu 16.04 desktop daily image ( xenial-desktop-amd64.iso ) from today. :(

[Ove Risberg (ove-risberg)]

The problem does not come from upstream... it it caused by a patch created by canonical.
debian/patches/00-fix_quote_readline_by_ref.patch:

From: JuanJo Ciarlante <email address hidden>
Subject: fix _quote_readline_by_ref to:
 - avoid escaping 1st '~' (lp: #1288314)
 - avoid quoting if empty, else expansion without args only shows dirs
   (lp: #1288031)
 - replace double escaping to single (eg for completing file/paths with
   spaces)
Origin: vendor, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=739835
Bug-Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=739835
Forwarded: yes, <email address hidden>

---
 bash_completion | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

--- bash-completion.orig/bash_completion
+++ bash-completion/bash_completion
@@ -536,13 +536,24 @@ __ltrim_colon_completions()
 # @param $2 Name of variable to return result to
 _quote_readline_by_ref()
 {
- if [[ $1 == \'* ]]; then
+ if [ -z "$1" ]; then
+ # avoid quoting if empty
+ printf -v $2 %s "$1"
+ elif [[ $1 == \'* ]]; then
         # Leave out first character
         printf -v $2 %s "${1:1}"
+ elif [[ $1 == ~* ]]; then
+ # avoid escaping first ~
+ printf -v $2 ~%q "${1:1}"
     else
         printf -v $2 %q "$1"
     fi

+ # Replace double escaping ( \\ ) by single ( \ )
+ # This happens always when argument is already escaped at cmdline,
+ # and passed to this function as e.g.: file\ with\ spaces
+ [[ ${!2} == *\\* ]] && printf -v $2 %s "${1//\\\\/\\}"
+
     # If result becomes quoted like this: $'string', re-evaluate in order to
     # drop the additional quoting. See also: http://www.mail-archive.com/
     # <email address hidden>/msg01942.html

[Ove Risberg (ove-risberg)]

I found this comment in lp: #1288314
https://bugs.launchpad.net/ubuntu/+source/bash-completion/+bug/1288031/comments/9
Ville Skyttä (vskytta) wrote on 2014-09-17: #9

FYI, the _quote_readline_by_ref changes should no longer be necessary with bash 4.3 patchlevel 10 and above, that's why they're not applied in upstream bash-completion. https://lists.gnu.org/archive/html/bug-bash/2014-04/msg00035.html

Please remove this bad patch from both trusty and xenial... or fix it...

[Andy Foster (andy-foster)]

Confirmed this problem still exists on 16.04 LTS Server.
After joining machine to AD using Winbind, logins with local accounts are noticeably slower and bash completion can be very laggy.
All these conflicting comments have confused me, is there actually a fix available?

[Ove Risberg (ove-risberg)]

The proper fix is to remove the "debian/patches/00-fix_quote_readline_by_ref.patch" patch from the bash-completion package.

While we wait for this to happen you can edit /usr/share/bash-completion/bash_completion and escape the ~ character on line 545 and 547 (on 14.04... on 16.04 it could be other line numbers).

This is really a big problem for Enterprise customers with slow Active Directory servers... and it is so easy to verify and fix.

[Launchpad Janitor (janitor)]

This bug was fixed in the package bash-completion - 1:2.1-4.2ubuntu2

---------------
bash-completion (1:2.1-4.2ubuntu2) yakkety; urgency=medium

  [Ove Risberg]
  * Add backslash for tilde character (LP: #1390061)
    - _quote_readline_by_ref on bash_completion
    - Adding backslash makes sure that
      tilde is not special character for bash, but character for purpose

 -- Seyeong Kim <email address hidden> Fri, 29 Apr 2016 10:55:55 +0900

[Chris J Arges (arges)]

Hello Ove, or anyone else affected,

Accepted bash-completion into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/bash-completion/1:2.1-4ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Can you fix the patch for Xenial and also include one for Wily? Thanks

[Seyeong Kim (seyeongkim)]

@arges

patch for xenail is already uploaded. could you please check that?

and I uploaded patch for wily,

Thanks

[Ove Risberg (ove-risberg)]

I installed bash-completion version 1:2.1-4ubuntu0.2 from trusty-proposed and the /usr/share/bash-completion/bash_completion file is identical to the one I have patched myself.

All my tests are successful... so I am happy :)

[Martin Pitt (pitti)]

Hello Ove, or anyone else affected,

Accepted bash-completion into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/bash-completion/1:2.1-4.2ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (seyeongkim)]

It works for me

ii bash-completion 1:2.1-4.2ubuntu1.1 all

[Launchpad Janitor (janitor)]

This bug was fixed in the package bash-completion - 1:2.1-4ubuntu0.2

---------------
bash-completion (1:2.1-4ubuntu0.2) trusty; urgency=medium

  [Ove Risberg]
  * Add backslash for tilde character (LP: #1390061)
    - _quote_readline_by_ref on bash_completion
    - Adding backslash makes sure that
      tilde is not special character for bash, but character for purpose

 -- Seyeong Kim <email address hidden> Fri, 29 Apr 2016 10:55:55 +0900

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for bash-completion has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Martin Pitt (pitti)]

Ove, can you please forward this to Debian too? This patch is in no way Ubuntu specific. Thanks!

[Ove Risberg (ove-risberg)]

Please check comment #12 and #13...
The patch debian/patches/00-fix_quote_readline_by_ref.patch is not from upstream so we do not have to forward this to Debian.

The proper fix for this is to remove the bad debian/patches/00-fix_quote_readline_by_ref.patch patch but since it took 1.5 years for someone to do anything about this I did not complain when they just patched the bad patch instead of removing the bad patch.

With this fix we can use Ubuntu in an Enterprise environment and Debian are not affected by this because the problem was caused by a Ubuntu patch.

[Launchpad Janitor (janitor)]

This bug was fixed in the package bash-completion - 1:2.1-4.2ubuntu1.1

---------------
bash-completion (1:2.1-4.2ubuntu1.1) xenial; urgency=medium

  [Ove Risberg]
  * Add backslash for tilde character (LP: #1390061)
    - _quote_readline_by_ref on bash_completion
    - Adding backslash makes sure that
      tilde is not special character for bash, but character for purpose

 -- Seyeong Kim <email address hidden> Fri, 29 Apr 2016 10:55:55 +0900

[Frank Hart (frank-hart)]

Hi Ove,

Thanks for the patch and your persistence in getting this fixed. We are also bitten by the bad 00-fix_quote_readline_by_ref.patch, but we are using Debian. So while the patch originated from Ubuntu, it has found its way into Debian as well. Would you like to submit a Debian bug about this? Otherwise, I can do it as well. And we agree that the proper way to fix this, is to just remove 00-fix_quote_readline_by_ref.patch.

[Ove Risberg (ove-risberg)]

Hi Frank,

Please report this to Debian... I am quite busy at work and at home at the moment so I do not have the time.

[Nathan Stratton Treadway (nathanst)]

Looks like this bug is covered by debbugs #825317, "bash-completion: Un-escaped "~*" leads to spurious NSS lookups"