dash as #!/bin/sh introduces countless incompatibilities

[comotion]

Binary package hint: dash

Hi,
I understand the rationale of using a lightweight bourne shell clone for #!/bin/sh instead of bash, however Ubuntu's use of dash as #!/bin/sh by default has broken countless scripts and brought us back into the dark ages of platform-specific scripting. Bashisms and various unixy #!/bin/sh variant quirks are well understood in their incomatabilites (especially by the autoconf/autopackage people), except this new player - dash.

Software vendors, open source projects and even the ubuntu BTS are flooded with bugs that stem from dash incompatabilites, which are often hard to debug the root cause of. Examples:
https://bugs.launchpad.net/ubuntu/+source/dash/+bug/92189
https://bugs.launchpad.net/ubuntu/+source/dash/+bug/105634
https://bugs.launchpad.net/ubuntu/+source/dash/+bug/71887
https://bugs.launchpad.net/ubuntu/+source/dash/+bug/135863
https://bugs.launchpad.net/ubuntu/+source/dash/+bug/128730

This is not a duplicate bug report - it the a compound of all the above bugs that
Examples of 3rd party apps that break:
the ATI installer, vmWare, Xen, Mathematica, NX Server, MKS, NVC and anything else that relies on a sane #!/bin/sh

The scope of this problem is rather large. It would be reasonable to say "fix the scripts" if the scripts were buggy, but alas this is not the case.

The following code snippets are extracted from live, working scripts that are used frequently, that work in other versions of #!/bin/sh, but break in dash:

$ dash -c 'cnt=6; while [ $((cnt = cnt - 1)) -gt 0 ]; do echo $cnt; done'
dash: arith: syntax error: "cnt = cnt - 1"

$ dash-c 'if false; then d="${foo/bar}"; fi'
    dash: Syntax error: Bad substitution

$ dash -c 'a=water;echo ${a:2:2}'
dash: Syntax error: Bad substitution

$ dash -c 'exec 20<&0'
exec: 1: 20: not found

$ dash -c 'echo $UID'
# no output. how on earth are we supposed to safely and portably get the UID?

$ dash -c 'echo hello 2>&1>/dev/null'
dash: Syntax error: redirection unexpected

$ dash -c 'x="1 2"; local y=$x; echo $y'
local: 1: 2: bad variable name

And an example from makeself:
shh.sh:
#!/bin/sh
tmpfile=foo
tmpdir=/tmp/dashtmp
mkdir "$tmpdir"
( cd "$tmpdir" && ( tar cvf - . | eval "gzip -c9" ) >> "$tmpfile" ) || { echo "Failed"; exit 1; }
echo >> "$tmpfile" >&-

$ ./shh.sh
./
./foo
./shh: line 6: echo: write error: Bad file descriptor

I invite others to collect similar examples here. If Ubuntu will not fix dashisms then we must support dash, which is why a compilation of incompatabilities is necessary, so that a list of workarounds can be put together, à la http://www.gnu.org/software/autoconf/manual/html_node/Portable-Shell.html, http://code.dogmap.org/lintsh/ and http://steve-parker.org/sh/sh.shtml which mostly cover outdated incompatabilites between other shells.

[William Grant (wgrant)]

The scripts *are* buggy. dash is POSIX-compliant, so all POSIX-compliant scripts will work with it. If they use bash-specific features, they must specify that they are to be executed by /bin/bash, rather than a general POSIX-compliant shell.

[comotion (comotion)]

I did not bring up POSIX compliance purposefully -I invite you to look at the links I provided to other bugs, which I feel discuss the POSIX compliance of dash adequately. The summary is that dash claims POSIX complainance, which is why dash is not fixed and continues to break valid scripts. I am not aware of any non-POSIXness in the examples above!

But that is neither here nor there- by always bringing up the POSIX strictness bit we're breaking with that time-honoured general rule: be lenient in what you accept conservative with what you generate. If this is a non-bug for ubuntu developers then it effectively becomes Everybody Else's Problem, - users and developers alike - as ubuntu is an otherwise excellent distribution which is really catching on with the general public.

Writing portable shell scripts is an art in itself. You cannot write a portable script by reading and knowing the POSIX sh standard. Much akin to writing browser-compatible websites, it requires you to know about all the specific issues that may arise in specific software, which again requires extensive first-hand experience. Dash introduces *new* quirkyness and *new* parsing issues that no other POSIX shell has troubles with, and does this in the name of POSIX compliance.

[Michael Zawrotny (zawrotny)]

POSIX compliance is exactly the heart of this. Just because dash *claims* to comply
with the standard doesn't necessarily mean that it *actually* does. Specifically, as quoted in bug #92189, section 2.6.4 of IEEE 1003.1 states

    If the shell variable x contains a value that forms a valid integer constant, then
    the arithmetic expansions "$((x))" and "$(($x))" shall return the same value.

If dash doesn't behave this way, which it doesn't, then it is not POSIX compliant. That being said, I don't believe that POSIX requires /bin/sh to be compliant. However, the whole trend in *nix systems over the last decade or two has been to _improve_ interoperability by adhering to standards. Using a non-POSIX shell for /bin/sh clearly goes against that trend. It seems pointless to introduce these compatibility issues that require either editing an enormous number of scripts or reconfiguring systems to symlink /bin/sh to bash instead of dash just to save a few kb of disk space.

In my opinion, that is doubly so since the incompatibility is recently introduced. That is, /bin/sh _used_ to be bash, and it was (IMO gratuitously) changed in a way that created problems. The maintainers should acknowledge the error and either patch dash to actually comply with POSIX or put bash back as the default /bin/sh.

[Paul Smith (psmith-gnu)]

First, we have to make a distinction between the issues described here which are POSIX and which mean that there is a bug in dash that should be fixed, and those which are NOT POSIX, of which, contrary to the assertion above, there are definitely some. For example:

$ dash -c 'a=water;echo ${a:2:2}'
dash: Syntax error: Bad substitution

is not POSIX: POSIX does not define substring syntax. Also:

$ dash -c 'echo $UID'
# no output. how on earth are we supposed to safely and portably get the UID?

POSIX does not define that the shell should set the UID variable. You have to use "id".

$ dash -c 'x="1 2"; local y=$x; echo $y'
local: 1: 2: bad variable name

I don't think "local" is defined in POSIX sh, although it appears dash does try to implement it; if so it should be done right.

$ dash -c 'cnt=6; while [ $((cnt = cnt - 1)) -gt 0 ]; do echo $cnt; done'
dash: arith: syntax error: "cnt = cnt - 1"

I don't think assignment inside arithmetic expressions is defined in POSIX.

The rest of these do appear to be bugs in dash, although technically the POSIX spec only guarantees the shell to handle file descriptors 0-9 so using any FD >9 is not, strictly, portable.

My opinion is that Ubuntu is 100% correct here: they should be using POSIX shell scripts and any script that does not adhere to POSIX should NOT start with #!/bin/sh. Obviously where dash doesn't support POSIX that's a bug in dash that needs to be fixed.

But, I hope Ubuntu will not bow down to this decade's version of "all the world's a VAX" and make more true the maxim "every /bin/sh is bash".

Fight the good fight, Ubuntu!

[Colin Watson (cjwatson)]

I have written https://wiki.ubuntu.com/DashAsBinSh (a new version, to replace the spec that used to be there) to address many of these issues, and linked it from the Edgy release notes.

As far as non-POSIX constructs go, Paul Smith is right on the money with his comments in https://bugs.launchpad.net/ubuntu/+source/dash/+bug/141481/comments/4, I think. The proper analogy is a C compiler: it is not a bug in the C compiler if it stops accepting code that doesn't conform to the standard and doesn't declare that it uses an extension (in this case, by saying #! /bin/bash). This is of course not to say that the real dash bugs that have been identified here should not be fixed.

[probono (probono)]

This is also being discussed on http://brainstorm.ubuntu.com/idea/2225/ - make your vote count ;-)

[neuromancer (neuromancer)]

Temporary tricks for running an sh scripts
http://liquidweather.net/howto/index.php?id=59

[Steve Parker (steve-steve-parker)]

Not really wanting to get into this debate, but having had my tutorial linked here and associated with an opinion, I must clarify that my ideal for UNIX and Linux systems is that /bin/sh be in some way consistent, and provide all Bourne functionality. That is, my personal view is that stating "#!/bin/sh" says that SVR4 sh is expected; scripts that assume bash should state "#!/bin/bash" (or possibly "#!/usr/bin/bash", which is a different issue).

That is academic; the point of this post is to refute the statement that "http://steve-parker.org/sh/sh.shtml ... mostly cover[s] outdated incompatabilites between other shells."

[ulin (uros-p-simic)]

At the very least, Ubuntu should make sure user's rc scripts (~/.profile, which sources ~/.bashrc for instance) does not contain dash-incompatible code by default, as it currently does. Programs like gdm source those but are run from dash resulting in user's stuff not being sourced before startup applications from ~/.config/autostart are lounched.

[Paul Smith (psmith-gnu)]

If GDM is sourcing the user's setup files using any shell other than the user's defined shell (from getpwent() or whatever) then that's a bug in GDM.

[Victor Mitin (caesarmv)]

There is one more DASH bug.

Described here:
http://groups.google.com/group/vim_use/browse_thread/thread/837988813a06c280/f8b4aadd575affaa?lnk=gst&q=expect#f8b4aadd575affaa