rpmio/set.c BYTE_ORDER patch for Solaris

[Tim Mooney]

Building rpm-5.4.12 on x86_64-pc-solaris2.11 (OpenIndiana oi151a8) with the Oracle Studio 12.3 compiler suite, though this issue would affect gcc too.

rpmio/set.c fails to compile because it's expecting BYTE_ORDER to be defined, as it would be on Linux with endian.h included:

        DEPDIR=.deps depmode=none /bin/sh ../depcomp \
        /bin/sh ../libtool --tag=CC --mode=compile cc -m64 -DHAVE_CONFIG_H -I. -I.. -I. -I.. -I../build -I../lib -I../lib -I../rpmdb -I../rpmio -I../misc -I../beecrypt/include -I../beecrypt/include -I../beecrypt -I../beecrypt -I../popt -I../popt -D__FUNCTION__=__func__ -I/local/gnu/include -I/local/include -I/local/include/neon -I/local/include/beecrypt -I/local/openssl/include -DRPM_OS_SOLARIS=021100 -I/local/include/db60 -I/usr/include/pcre -xopenmp -Xa -xc99=all -xO3 -mt -KPIC -D__FUNCTION__=__func__ -m64 -I/local/gnu/include -I/local/include -I/local/include/neon -I/local/include/beecrypt -I/local/openssl/include -D_GNU_SOURCE -D_REENTRANT -c -o set.lo set.c
libtool: compile: cc -m64 -DHAVE_CONFIG_H -I. -I.. -I. -I.. -I../build -I../lib -I../lib -I../rpmdb -I../rpmio -I../misc -I../beecrypt/include -I../beecrypt/include -I../beecrypt -I../beecrypt -I../popt -I../popt -D__FUNCTION__=__func__ -I/local/gnu/include -I/local/include -I/local/include/neon -I/local/include/beecrypt -I/local/openssl/include -DRPM_OS_SOLARIS=021100 -I/local/include/db60 -I/usr/include/pcre -xopenmp -Xa -xc99=all -xO3 -mt -KPIC -D__FUNCTION__=__func__ -m64 -I/local/gnu/include -I/local/include -I/local/include/neon -I/local/include/beecrypt -I/local/openssl/include -D_GNU_SOURCE -D_REENTRANT -c set.c -KPIC -DPIC -o .libs/set.o
"set.c", line 128: syntax error before or at: ...
"set.c", line 448: #error: "unknown byte order"
cc: acomp failed for set.c
gmake[4]: *** [set.lo] Error 1
gmake[4]: Leaving directory `/local/src/RPM/BUILD/rpm-5.4.12/rpmio'

Note there are two syntax errors there -- I'll tackle the more difficult issue from line 128 (and elsewhere) in a separate report.

Solaris doesn't have endian.h. It has sys/byteorder.h (which actually includes sys/isa_defs.h for the important stuff). Including sys/byteorder.h gets you either _LITTLE_ENDIAN or _BIG_ENDIAN defined (to nothing). From there, it's trivial to make a set of defines that are Linux-compatible. I've had to use essentially the same patch for ruby and a few other programs that expect the BSD and Linux BYTEORDER and LITTLE_ENDIAN symbols to be defined.

The attached patch adds a configure check for sys/byteorder.h and code in system.h to rig up BYTE_ORDER, LITTLE_ENDIAN,
etc. in system.h.

NOTE: I notice that configure.ac is checking for sys/endian.h but it's not checking for just "endian.h". None of the Linux systems I have access to have a sys/endian.h, but they do have endian.h and bits/endian.h. It seems to me like the only reason the current byte order dependent code is working even for Linux is because sys/types.h eventually includes endian.h.
Ideally the ifdef guards I added in system.h would check for !defined(HAVE_ENDIAN_H), but because that's not currently being checked for I didn't add it.

If you would prefer the BYTE_ORDER defines be directly in set.c, I can provide a new patch. Alternately, it looks like set.c is derived from elsewhere. If you can provide me with where upstream is, I would be happy to report the issue to upstream.

[Jeff Johnson (n3npq)]

Thanks for the portable endian.h retrofit: there are likely lots
of additional issues just fixed in the NIST SHA3 digests that just got
fixed as well.

There's a number of tricky non-portable GNUisms in set.c, including " … "
initialization as well as the remaining nested static function in need of fixing.

See the -DSELF_TEST which absolutely MUST pass for any serious refactoring
of the rpmio/set.c code: the algorithm is quite complex, debugging any serious
breakage would take a week or more to figure out.