True CREATED return value for ENSURE-DIRECTORIES-EXIST when multiple processes attempt to create the same directory.

[Mark Cox]

ENSURE-DIRECTORIES-EXIST will return a true value for CREATED when another process successfully created the directory during the invocation of ENSURE-DIRECTORIES-EXIST. Example code is shown below:

(defun test-loop ()
  (let ((pathname #P"/tmp/example/"))
    (loop
      for attempt from 0 below 100
      until
      (progn
        (ignore-errors (sb-posix:rmdir pathname))
        (let* ((t1-created nil)
               (t2-created nil)
               (t1 (sb-thread:make-thread (lambda ()
                                            (setf t1-created (nth-value 1 (ensure-directories-exist pathname))))))
               (t2 (sb-thread:make-thread (lambda ()
                                            (setf t2-created (nth-value 1 (ensure-directories-exist pathname)))))))
          (sb-thread:join-thread t1)
          (sb-thread:join-thread t2)
          (cond ((and t1-created t2-created)
                 (format t "~&Both threads created the directory ~A on attempt ~d.~%" pathname attempt)
                 t)
                ((or t1-created t2-created)
                 nil)
                ((and (null t1-created) (null t2-created))
                 (format t "~&Neither thread created the directory ~A on attempt ~d.~%" pathname attempt)
                 nil)))))))

$ sbcl --noinform --no-userinit --non-interactive --load /tmp/test.lisp --eval '(test-loop)'
Both threads created the directory /tmp/example/ on attempt 0.

It is expected that only one invocation of ENSURE-DIRECTORIES-EXIST would create the directory. The Hypserspec entry for ENSURE-DIRECTORIES-EXIST states

   The secondary value, created, is true if any directories were created.

which I interpret as ENSURE-DIRECTORIES-EXIST only returns true if the invocation of ENSURE-DIRECTORIES-EXIST creates the directory.

The implementation of ENSURE-DIRECTORIES-EXIST invokes SB-UNIX:UNIX-MKDIR without checking the errno value for SB-UNIX:EEXIST.

It is possible that the message printed by ENSURE-DIRECTORIES-EXIST when VERBOSE is true might need to change. The Hyperspec entry for ENSURE-DIRECTORIES-EXIST states

  If the containing directories do not exist and if verbose is true, then the
  implementation is permitted (but not required) to perform output to standard
  output saying what directories were created.

meaning that a message could be printed after any directories were created. Currently, messages are printed prior to creation which may result in two processes outputting a message when only one succeeds.

The requested behaviour is useful for ensuring that the created directory is new e.g. temporary directories.

sbcl --version:
SBCL 2.4.7

uname -a:
22.6.0 Darwin Kernel Version 22.6.0: Mon Jun 24 01:25:37 PDT 2024; root:xnu-8796.141.3.706.2~1/RELEASE_X86_64 x86_64

*FEATURES*:
(:ARENA-ALLOCATOR :X86-64 :GENCGC :64-BIT :ANSI-CL :BSD :COMMON-LISP :DARWIN
 :IEEE-FLOATING-POINT :LITTLE-ENDIAN :MACH-O :PACKAGE-LOCAL-NICKNAMES
 :SB-CORE-COMPRESSION :SB-LDB :SB-PACKAGE-LOCKS :SB-THREAD :SB-UNICODE :SBCL
 :UNIX)

[Gábor Melis (melisgl)]

This all makes sense. Looking at the code, there is a comment that hints at historical(?) complications: "If we can trust mkdir returning ENOENT and EEXIST". Something to be checked I guess. This is the point where NFS might enter the discussion. Also note that Windows CreateDirectory hides behind UNIX-MKDIR and the error codes need to match up.