network tests do not follow normal testing policy

[Jeff Johnson]

This issue records a situation that emerged during investigation of a
specific bug. The issue itself is rather non-specific, and so mostly
just provides a way to reference the situation.

The tests in the network section of LSB.os (at least) have a lot of
places where they detect failure conditions, but don't do the right
thing with them. What should happen, roughly, is that a test should be
run until it generates sufficient information to be classed as PASS
or FAIL. However, to get to the test steps, there is often some setup
needed, and things could fail here as well. If this happens, the test
often needs to be abandoned. In this case it should be reported as
UNRESOLVED, because we did not ever get to the place where we could
decide whether the feature under test passed or failed. Here's an
example to illustrate (compressed slightly from the actual source to fit
more on the screen):

/* This test case checks accept for return value -1.
 * If socket flag is set to O_NONBLOCK and there are no connections are present
 * to be accepted, errno is set to EAGAIN or EWOULDBLOCK. */
private void test1()
{
    int flags;
    tet_infoline("Checks accept for errno value EAGAIN or EWOULDBLOCK");

    if((sockfd=socket(AF_INET,SOCK_STREAM,0)) == -1) {
        tet_printf("socket syscall returned error");
1 return;
    }

    if((flags = fcntl(sockfd, F_GETFL)) == -1) {
        tet_printf("fcntl syscall returned error on F_GETFL");
2 return;
    }
    flags |= O_NONBLOCK;

    if((fcntl(sockfd, F_SETFL, flags)) == -1) {
        tet_printf("fcntl syscall returned error on F_SETFL");
3 return;
    }

    myaddr.sin_family=AF_INET;
    myaddr.sin_port=htons(0);
    myaddr.sin_addr.s_addr=htonl(INADDR_ANY);
    if ((bind(sockfd,(struct sockaddr *)&myaddr,sizeof(myaddr))) == -1) {
        tet_printf("bind syscall returned error");
4 return;
    }

    if((listen(sockfd,0)) == -1) {
        tet_printf("listen syscall returned error");
5 return;
    }

    errno=0;
    if((ret=accept(sockfd,NULL,NULL)) != -1) {
        tet_printf("accept() returned %d, expected to return -1", ret);
        tet_result(TET_FAIL);
    } else {
        err=errno;
        if (err != EAGAIN || err != EWOULDBLOCK) {
        tet_printf("errno is set to %d, it is expected to be %d or %d [EAGAIN
or EWOULDBLOCK]", err,EAGAIN,EWOULDBLOCK);
        tet_result(TET_FAIL);
        } else {
        tet_result(TET_PASS);
        }
    }
    shutdown(sockfd,SHUT_RDWR);
}

At lines marked 1,2,3,4,5 we print a message to the tet journal and
abort the test by returning, but in doing so, we have not set a result
code for the test. On generating a report, this will end up showing
as UNREPORTED, which is not a good thing to report, when we actually
saw the error condition and could report on it in a somewhat more
useful way. In addition, if the socket is established and there is a
subsequent fail before getting to the actual test, which is for accept()
functionality, we don't even shut down the socket.

As a contrast, here's some of the original vsx POSIX code. It depends
heavily on macros, but the difference should show anyway:

        switch (tty_prep(&tty_fildes, tty_termios_p))
        {
        case 0:
                PATH_TRACE;
                break;
        case 1:
1 xx_rpt(UNSUPPORTED);
                in_rpt("VSX_TERMIOS_TTY is set to \"unsup\"");
                DBUG_VOID_RETURN;
        default:
2 xx_rpt(DELETION);
                in_rpt("open/initialise of VSX_TERMIOS_TTY failed");
                DBUG_VOID_RETURN;
        }

        tty_termios_p->c_cc[VSTART] = 0;
        tty_termios_p->c_cc[VSTOP] = 0;

        if (tcgetattr(tty_fildes, tty_termios_p) == SYSERROR) {
                err = errno;
3 xx_rpt(DELETION);
                in_rpt("tcgetattr() failed on VSX_TERMIOS_TTY - errno %d (%s)",
                        err, errname(err));
                DBUG_VOID_RETURN;
        }
        else
                PATH_TRACE;

Here the lines marked 1,2,3 are problems nor directly under test,
and they're handled by setting the result (by xx_rpt) to DELETION
(translates to UNRESOLVED) if a setup step failed, or to UNSUPPORTED if
feature for the test, as described in the config file, is not available,
then the message is emitted (in_rpt call) before returning (DBUG_VOID_RETURN)

We have some places that are even worse, where the test just plows on
after seeing and reporting an error:

/* This test case checks recv for error EAGAIN or EWOULDBLOCK.
 * This is done by setting O_NONBLOCK flag for fd pointed by
 * 'socket' argument. Expected return value is EAGAIN or
 * EWOULDBLOCK. */
private void
test7()
{
    int flags;
    struct sockaddr_un srv_addr;

    sockfd=socket(AF_UNIX,SOCK_DGRAM,0);

    if (!(sockfd >= 1)) {
        tet_printf("Error in creating socket");
1 return;
    }

    srv_addr.sun_family = AF_UNIX;
    strcpy(srv_addr.sun_path, "server");
    if (bind(sockfd, (struct sockaddr *) &srv_addr, sizeof(srv_addr)) < 0) {
        tet_printf("ERROR on binding\n");
2 return;
    }

    /* Set a socket as nonblocking */
    if((flags = fcntl(sockfd, F_GETFL)) < 0)
3 tet_printf("fcntl F_GETFL error\n");
    flags |= O_NONBLOCK;
    if(fcntl(sockfd, F_SETFL, flags) < 0)
4 tet_printf("fcntl F_SETFL error\n");

Here cases 1,2 are as in the first snippet, but cases 3,4 can see an
error that should abort the test but do not.