Comment 245 for bug 554172

First some background. Analysis above indicates that upstart jobs are
failing because opens of /dev/console are failing, and that this should
not be possible:

 "... open("/dev/console") is _not_supposed_to_fail_."

As previously indicated /dev/console is a virtual device which represents
the currently active console. However, there is no guarentee that this
device will successfully open, no guarentee that there is a system
console device. This device will only open successfully if there is
a real active system console defined, if there is no system console at
open time then the open will fail with errno set to ENODEV. There is no
guarentee that it will ever become openable.

Also while /dev/console is a virtual device representing the current active
console, it is implemented as a direct open of the real console device.
Opens of /dev/console are redirected to opens of the actual active console
device at the time the open occurs. The open is therefore to the real
underlying device, the returned file descriptor has all the semantics of
the real tty device.

I have managed to trivially reproduce open failures to /dev/console,
returning errno of EIO, by running two open/close loops on a tty device
in parallel, in my case /dev/tty10:

    #include <stdio.h>
    #include <fcntl.h>
    #include <errno.h>

    main(int argc, char *argv[])
    {
     int fd;

     while (1) {
      fd = open(argv[1], O_RDWR);
      if (fd < 0) {
       printf("fd<%d> errno<%d>\n", fd, errno);
       fflush(stdout);
      }
      close(fd);
     }
    }

Looking at the open code, the suspected source of the error (EIO during open)
is the code fragment below. That the console is in the process of being
closed when the open is occuring:

    static int tty_reopen(struct tty_struct *tty)
    {
     struct tty_driver *driver = tty->driver;

     if (test_bit(TTY_CLOSING, &tty->flags))
      return -EIO;
    [...]

Nominally open/close handling is protected by tty_mutex, this prevents
parallel opens and closes from racing with each other. However once we
close a device for the last time (ie all sharers have closed the device)
a real shutdown of the device occurs. For tty devices this may involve
an extended handshake at the hardware level. During this close process
is it not safe to initiate a reopen, but we also do not wish to block all
tty opens. Thus the kernel only holds the tty_mutex long enough to mark
the device as in the process of closing (sets TTY_CLOSING in the device
flags) and releases tty_mutex before executing the potentially extended
close handling. In the single thread close/reopen race is avoided as tty
shutdown processing is executed in the context of the closer, thus the
device close has progressed sufficiently far to prevent a subsequent open
from seeing this partially closed state, and triggering the EIO return.
It should be noted that open/close processing is also covered by the BKL,
however this is not proof against parallel execution, should the close
handing sleep (which can occur should there be any mutexs in the path)
the BKL is dropped and reaquired (as it is a preemptable lock). For the
common case the console is a VT device, which needs to take console_sem
during device shutdown which opens the race with another parallel thread.

This EIO return has been pointed out to the tty maintainers, but in short
this is deemed to be correct behaviour, though it is not well documented.
The window during which this was possible was reduced significantly by
splitting close processing, but the window is unlikely be removed, as
the behaviour has meaning, to quote from the git log:

      "Fun but it's actually not a bug and the fix is wrong in itself as
      the port may be closing but not yet being destructed, in which case
      it seems to do the wrong thing. Opening a tty that is closing (and
      could be closing for long periods) is supposed to return -EIO."

In short the EIO behaviour looks to be expected behavior on a real TTY
device, also /dev/console cannot be expected to always open, but must
be expected to have the same semantics as the underlying active console
device.

Looking forward, there is much activity in the upstream kernel in
the area of removing the BKL which currently does impact on this area.
There are currently patches proposed for (but not merged to) v2.6.36 which
substitute the BKL for a tty specific mutex. This appears that it will
avoid this open/close race by rendering all opens and closes serialised.
It is not yet clear that this change in semantics was planned as the
code to handle overlapping open/closes has not been removed. We need
to understand if this change in semantics was intended before we could
consider changing the semantics for Maverick or Lucid.

For Lucid we should also consider the difficulty in testing any change of
this nature, as it covers a large range of hardware devices we are unable
test. Therefore it seems most appropriate to handle this EIO return in
userspace, which in general is already handling similar returns such as
EINTR and retrying the open.

Recommendations:

1) any consumer of the /dev/console must handle its failure to open where
there is no console available, ENODEV is returned from open.
2) any consumer of /dev/console should expect its failure to open returning
EIO, and should retry the open.