posix-environ borks on phantom entry

[Faré]

While developing XCVB, I was _sometimes_ experiencing weird errors while running run-program. I narrowed them to this:

CL-USER> (sb-impl::posix-environ)
c-string decoding error (:external-format :UTF-8):
  the octet sequence 2 cannot be decoded.
   [Condition of type SB-INT:C-STRING-DECODING-ERROR]

Restarts:
 0: [RETRY] Retry SLIME REPL evaluation request.
 1: [ABORT] Return to SLIME's top level.
 2: [ABORT] Exit debugger, returning to top level.

Backtrace:
  0: (SB-INT:C-STRING-DECODING-ERROR :UTF-8 2)
  1: (SB-IMPL::READ-FROM-C-STRING/UTF-8 #.(SB-SYS:INT-SAP #X10038CD8A0) CHARACTER)
  2: (SB-INT:C-STRINGS->STRING-LIST #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X7FFFFFFFDE30 :TYPE (* (C-STRING :EXTERNAL-FORMAT :DEFAULT :ELEMENT-TYPE CHARACTER))>)
  3: (SB-INT:SIMPLE-EVAL-IN-LEXENV (POSIX-ENVIRON) #<NULL-LEXENV>)

If I try to force the latin1 external format instead of utf-8, I get the following:

CL-USER> (let ((SB-ALIEN::*DEFAULT-C-STRING-EXTERNAL-FORMAT* :latin1)) (sb-impl::posix-environ))
("PWD=/home/fare/cl/xcvb" "DISPLAY=:0" "TERM=dumb" "TERMCAP=" "COLUMNS=100" "EMACS=t" "INSIDE_EMACS=23.1.50.1,comint" "WHERE=B612" "found=mutt" "NETHACKOPTIONS=boulder=$,color,DECgraphics,!sparkle,hilite_pet" "XCVB_PATH=/home/fare/cl:/home/fare/.local/share/common-lisp" "EDITFILES=" "COLORTERM=y" "tty=/dev/pts/13" "BASESHLVL=2" "LOGLVL=2" "LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=01;33:so=01;35:bd=40;33;01:cd=40;33;01:ex=01;32:mi=00:or=01;36:lc=\\e[:rc=m:ec=\\e[00m:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.tar=01;31:*.tgz=01;31:*.tbz=01;31:*.tb2=01;31:*.ttz=01;31:*.taz=01;31:*.tpz=01;31:*.arj=01;31:*.zoo=01;31:*.lzh=01;31:*.zip=01;31:*.bz=01;31:*.b2=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz=01;31:*.bz2=01;31:*.tz=01;31:*.psz=01;31:*.txz=01;31:*.jpg=01;35:*.gif=01;35:*.jpeg=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35" "BROWSER=" "BACKUPDIR=/tmp/fare/backup" "EDITOR_ENC=utf-8" "METAMAIL_PAGER=/usr/bin/less" "MM_PAGER=/usr/bin/less" "MANPAGER=/usr/bin/less -s" "EDITOR=/home/fare/bin/emacs.sh" "MAILREADER=/usr/bin/mutt" "PAGER=/usr/bin/less" "BINDIR=/home/fare/bin" "REPLYTO=Francois-Rene Rideau <email address hidden>" "SLATE_ROOT=/home/fare/src/slate/main" "NICKNAME=Fare" "IRCNAME=http://fare.tunes.org/" "WWW_HOME=file:/home/fare/etc/www/index.html" "FARE=/home/fare" "FORTUNE=/usr/games/fortune" "XLIBDIR=/usr/X11R6/lib/X11" "MAILCAP=/etc/mailcap" "INFOPATH=/usr/share/info:/usr/info:/usr/local/info" "TEXINPUTS=.:/home/fare/fare/texstuff:/usr/local/share/texmf//:" "MYFORTUNES=/usr/local/share/games/fortunes/fare" "FORTUNES=/usr/local/share/games/fortunes" "MYSRC=/usr/src" "TUNESRC=/home/fare/etc/tunes/B612" "TUNES=/usr/src/tunes" "CVS_RSH=ssh" "RSYNC_RSH=ssh -x" "TMP=/tmp/fare" "MYTMP=/tmp/fare" "DOWNLOAD_DIR=/DL" "BACKSPACE=" "LESS=-fMMQQi" "BINTYPE=x86_64" "MANPATH=/usr/local/man:/usr/share/man:/usr/man:/usr/X11R6/man" "MYETC=/home/fare/etc" "LOCALSYSTEM=B612" "ORIG_PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games" "ZDOTDIR=/home/fare" "OLDPWD=/home/fare" "WINDOWID=50331671" "KONSOLE_DBUS_SESSION=/Sessions/1" "KONSOLE_DBUS_SERVICE=:1.33" "COLORFGBG=15;0" "PROFILEHOME=" "LANGUAGE=" "DESKTOP_STARTUP_ID=" "GTK2_RC_FILES=/etc/gtk-2.0/gtkrc:/home/fare/.gtkrc-2.0::/home/fare/.kde/share/config/gtkrc-2.0" "GTK_RC_FILES=/etc/gtk/gtkrc:/home/fare/.gtkrc::/home/fare/.kde/share/config/gtkrc" "SESSION_MANAGER=local/B612:@/tmp/.ICE-unix/5558,unix/B612:/tmp/.ICE-unix/5558" "KDE_MULTIHEAD=false" "_=/usr/bin/detachtty" "XAUTHORITY=/home/fare/.Xauthority" "GTK_IM_MODULE=scim-bridge" "QT_PLUGIN_PATH=/home/fare/.kde/lib/kde4/plugins/:/usr/lib/kde4/plugins/" "WINDOWPATH=7" "LC_CTYPE=en_US.UTF-8" "XDG_DATA_DIRS=/usr/local/share/:/usr/share/:/usr/share/gdm/" "DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-WWmeHlsB8N,guid=6fbad145130d800cad97f5aa4ae1b90d" "LOGNAME=fare" "XCURSOR_THEME=Oxygen_Black" "KDE_SESSION_VERSION=4" "SHLVL=4" "HOME=/home/fare" "GDMSESSION=kde" "GDM_LANG=en_US.UTF-8" "LANG=en_US.UTF-8" "KDE_SESSION_UID=1000" "XMODIFIERS=@im=SCIM" "GDM_XSERVER_LOCATION=local" "QT_IM_MODULE=scim-bridge" "PATH=/home/fare/bin/x86_64:/home/fare/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/local/bin/x86_64:/local/bin:/usr/local/sbin:/usr/sbin:/sbin:/conf/generic:/usr/local/games/bin:/usr/games:/usr/games/X11:/usr/games" "DESKTOP_SESSION=kde" "USERNAME=fare" "SSH_AUTH_SOCK=/tmp/ssh-xpKIZs5320/agent.5320" "USER=fare" "LC_ALL=en_US.UTF-8" "KDE_FULL_SESSION=true" "GS_LIB=/home/fare/.fonts" "SSH_AGENT_PID=5410" "SHELL=/bin/zsh" "WINDOW=0" "STY=6491.X" "Ð")

Notice the last entry, which is a string with a single character #\xD0, possibly an artefact of SBCL being confused as to where the environment ends.

/usr/bin/env, when called from emacs or the shell, does not have this invalid environment entry (invalid because it doesn't even have a = delimiter).

For reference, I get this behavior on SBCL 1.0.31.32 on Linux amd64.

[Nikodemus Siivola (nikodemus)]

Can you provide a test-case that sets up an environment where SBCL gets a bogus entry like this?

[Faré (fahree)]

If I try at the command line, I can't reproduce on demand, but if I remember correctly, was getting the error once in a while when running batch tests of XCVB before I inserted my workaround.

However, it pretty much happens to me almost all the time when running under SLIME. In a same run, I just got " ", a series of "4" then a series of "Z^K" (that's two characters, Z and Control-K) as the phantom entry, and finally a c-string decoding error (:external-format :UTF-8). So it varies during a same run.

Running 1.0.32.30 + random patch on linux/amd64 with SLIME 2009-11-06 on Emacs 23.1.50.1 from debian.

It may or may not be of note that I run in a UTF-8 environment and that SLIME somehow specifies :coding-system "iso-latin-1-unix" in its call to swank::start-server.

If you have trouble reproducing at home, I can try to get you access to a machine where I can reproduce the bug (which is even easier if you have access to systems at my current employer's).

[Faré (fahree)]

OK, so I disabled my workaroud and tried again until after a few attempts I got a failure again.

Backtrace:
  0: (SB-INT:C-STRING-DECODING-ERROR :UTF-8 1)
  1: (SB-IMPL::READ-FROM-C-STRING/UTF-8 #<system area pointer: #X10036C1C50> CHARACTER)
  2: (SB-INT:C-STRINGS->STRING-LIST #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X7FFFFFFFDE30 :TYPE (* (C-STRING :EXTERNAL-FORMAT :DEFAULT :ELEMENT-TYPE CHARACTER))>)

* (sb-impl::wrapped-environ)
#<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP #X7FFFFFFFDE30 :TYPE (* (C-STRING :EXTERNAL-FORMAT :DEFAULT :ELEMENT-TYPE CHARACTER))>
* (format t "~&~{ ~16,'0X~%~}~&" (loop :for i :below 102 :collect (sb-sys:sap-ref-64 (sb-sys:int-sap #X7FFFFFFFDE30) (* i 8))))
 00007FFFFFFFE2C7
 00007FFFFFFFE2D6
 00007FFFFFFFE2E1
...
 00007FFFFFFFEFA8
 00007FFFFFFFEFB7
 00007FFFFFFFEFC0
 00000010036C1C50
 0000000000000000
NIL
* (coerce (loop :for i :below 30 :collect (code-char (sb-sys:sap-ref-8 (sb-sys:int-sap #x00000010036C1C50) i))) 'string)
"'2o^C^P^@^@^@^W^@^P ^@^@^@^@ª^@^@^@^@^@^@^@`^@^@^@^@^@"

My working hypothesis is that my workaround to a previous problem caused this. In XCVB, I have:
main.lisp: #+sbcl (sb-posix:putenv (strcat "SBCL_HOME=" *lisp-implementation-directory*))
and when I query (sb-ext:posix-environ) early enough the last entry is indeed "SBCL_HOME=/usr/local/lib/sbcl/".
But at one time or another, garbage collection happens and moved the string (what else but GC would have such effect?).

If this hypothesis is correct, then the bug is with putenv using a link to a Lisp-heap-allocated string instead of using malloc and copying the string to a safe stable place before to add it to the environment.

[Faré (fahree)]

Note that setenv and unsetenv seem to be better interfaces to provide than putenv, and do the malloc'ing for you. It would be nice if instead of fixing putenv (that OBVIOUSLY no one is using), sb-posix would instead just remove it and add setenv / unsetenv in its place.

[Faré (fahree)]

Manually tested to work and survive gc on sbcl 1.0.32.30 amd64, unlike putenv.

[Nikodemus Siivola (nikodemus)]

  status fixcommitted

Thanks! Fixed in 1.0.33.21.

[Faré (fahree)]

Woohoo! Thanks a lot.