Build problems with MPICH2 (and similar) MPI libraries

[Jens]

configuring DOLFIN with icc failes to build with scotch because SCOTCH_TEST_RUNS failed
because mpi.h is included after stdio.h in scotch_test_lib.cpp

I get
impi/4.0.2.003/include64/mpicxx.h(45): catastrophic error: #error directive: "SEEK_SET is #defined but must not be for the C++ binding of MPI. Include mpi.h before stdio.h"

full (relevant) log with "-DSCOTCH_DEBUG:BOOL=ON"

-- Performing test SCOTCH_TEST_RUNS
-- Performing test SCOTCH_TEST_RUNS - Failed
-- [ /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/cmake/modules/FindScotchPT.cmake:171 ] SCOTCH_TEST_LIB_COMPILED = FALSE
-- [ /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/cmake/modules/FindScotchPT.cmake:173 ] SCOTCH_TEST_LIB_COMPILE_OUTPUT = Change Dir: /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/CMakeTmp

Run Build Command:/usr/bin/gmake "cmTryCompileExec/fast"
/usr/bin/gmake -f CMakeFiles/cmTryCompileExec.dir/build.make CMakeFiles/cmTryCompileExec.dir/build
gmake[1]: Entering directory `/tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/CMakeTmp'
/vscmnt/gent_gulpin/_apps_gent/gengar/harpertown/software/CMake/2.8.4-ictce-4.0.6/bin/cmake -E cmake_progress_report /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/CMakeTmp/CMakeFiles 1
Building CXX object CMakeFiles/cmTryCompileExec.dir/scotch_test_lib.cpp.o
/apps/gent/gengar/harpertown/software/impi/4.0.2.003/bin64/mpiicpc -xHOST -O2 -I/apps/gent/gengar/harpertown/software/SCOTCH/5.1.12b_esmumps-ictce-4.0.6/include -I/apps/gent/gengar/harpertown/software/impi/4.0.2.003/include64 -o CMakeFiles/cmTryCompileExec.dir/scotch_test_lib.cpp.o -c /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/scotch_test_lib.cpp
/apps/gent/gengar/harpertown/software/impi/4.0.2.003/include64/mpicxx.h(45): catastrophic error: #error directive: "SEEK_SET is #defined but must not be for the C++ binding of MPI. Include mpi.h before stdio.h"
  #error "SEEK_SET is #defined but must not be for the C++ binding of MPI. Include mpi.h before stdio.h"
   ^

compilation aborted for /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/scotch_test_lib.cpp (code 4)
gmake[1]: *** [CMakeFiles/cmTryCompileExec.dir/scotch_test_lib.cpp.o] Error 4
gmake[1]: Leaving directory `/tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/CMakeTmp'
gmake: *** [cmTryCompileExec/fast] Error 2

-- [ /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/cmake/modules/FindScotchPT.cmake:175 ] SCOTCH_TEST_LIB_EXITCODE =
-- [ /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/cmake/modules/FindScotchPT.cmake:177 ] SCOTCH_TEST_LIB_OUTPUT =
-- Found ZLIB: /usr/include (found version "1.2.3")
-- Performing test SCOTCH_ZLIB_TEST_RUNS
-- Performing test SCOTCH_ZLIB_TEST_RUNS - Failed
-- [ /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/cmake/modules/FindScotchPT.cmake:214 ] SCOTCH_ZLIB_TEST_LIB_COMPILED = FALSE
-- [ /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/cmake/modules/FindScotchPT.cmake:216 ] SCOTCH_ZLIB_TEST_LIB_COMPILE_OUTPUT = Change Dir: /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/CMakeTmp

Run Build Command:/usr/bin/gmake "cmTryCompileExec/fast"
/usr/bin/gmake -f CMakeFiles/cmTryCompileExec.dir/build.make CMakeFiles/cmTryCompileExec.dir/build
gmake[1]: Entering directory `/tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/CMakeTmp'
/vscmnt/gent_gulpin/_apps_gent/gengar/harpertown/software/CMake/2.8.4-ictce-4.0.6/bin/cmake -E cmake_progress_report /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/CMakeTmp/CMakeFiles 1
Building CXX object CMakeFiles/cmTryCompileExec.dir/scotch_test_lib.cpp.o
/apps/gent/gengar/harpertown/software/impi/4.0.2.003/bin64/mpiicpc -xHOST -O2 -I/apps/gent/gengar/harpertown/software/SCOTCH/5.1.12b_esmumps-ictce-4.0.6/include -I/apps/gent/gengar/harpertown/software/impi/4.0.2.003/include64 -o CMakeFiles/cmTryCompileExec.dir/scotch_test_lib.cpp.o -c /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/scotch_test_lib.cpp
/apps/gent/gengar/harpertown/software/impi/4.0.2.003/include64/mpicxx.h(45): catastrophic error: #error directive: "SEEK_SET is #defined but must not be for the C++ binding of MPI. Include mpi.h before stdio.h"
  #error "SEEK_SET is #defined but must not be for the C++ binding of MPI. Include mpi.h before stdio.h"
   ^

compilation aborted for /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/scotch_test_lib.cpp (code 4)
gmake[1]: *** [CMakeFiles/cmTryCompileExec.dir/scotch_test_lib.cpp.o] Error 4
gmake[1]: Leaving directory `/tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/CMakeTmp'
gmake: *** [cmTryCompileExec/fast] Error 2

[Jens (jens.timmerman)]

to clarify, this is when building dolfin-1.0.0

[Garth Wells (garth-wells)]

I suspect that this may not be an icc issue. Your are probablt using the Intel MPI, which seems to be based on mpich. mpich needs a special flag to build with c++.

[Jens (jens.timmerman)]

Yes, I am using Intel MPI with icc...

see the actuall compilation:
So I am not using c++ anywhere here.
/apps/gent/gengar/harpertown/software/impi/4.0.2.003/bin64/mpiicpc -xHOST -O2 -I/apps/gent/gengar/harpertown/software/SCOTCH/5.1.12b_esmumps-ictce-4.0.6/include -I/apps/gent/gengar/harpertown/software/impi/4.0.2.003/include64 -o CMakeFiles/cmTryCompileExec.dir/scotch_test_lib.cpp.o -c /tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/scotch_test_lib.cpp

[Garth Wells (garth-wells)]

From the MPICH2 user guide (http://www.mcs.anl.gov/research/projects/mpich2/documentation/files/mpich2-1.4.1-userguide.pdf)

4.1 Special Issues for C++
Some users may get error messages such as
SEEK_SET is #defined but must not be for the C++ binding of MPI
The problem is that both stdio.h and the MPI C++ interface use SEEK SET,
SEEK CUR, and SEEK END. This is really a bug in the MPI-2 standard. You
can try adding
#undef SEEK_SET
#undef SEEK_END
#undef SEEK_CUR
before mpi.h is included, or add the definition
-DMPICH_IGNORE_CXX_SEEK
to the command line (this will cause the MPI versions of SEEK SET etc. to
be skipped)

[Jens (jens.timmerman)]

Yes, this is exactly the behavior I'm trying to report.
the SCOTCH_TEST_RUNS includes mpi.h after stdio.h and does not have -DMPICH_IGNORE_CXX_SEEK set.

Could this be fixed?
Or is there an easy way for me to dissable this test run?

[Jens (jens.timmerman)]

I'm sorry, I misread and typed c++ as g++

[Johannes Ring (johannr)]

Please see if the attached patch fixes this issue.

[Garth Wells (garth-wells)]

On 26 June 2012 08:42, Johannes Ring <email address hidden> wrote:
> Please see if the attached patch fixes this issue.
>

I don't think that this is the right solution. We should make sure
that MPI detection gets the right flags, which should in turn be used
in the configuration tests.

Garth

> ** Patch added: "scotch_test.patch"
>   https://bugs.launchpad.net/dolfin/+bug/1016540/+attachment/3204549/+files/scotch_test.patch
>
> ** Changed in: dolfin
>    Milestone: None => 1.0.1
>
> --
> You received this bug notification because you are a member of DOLFIN
> Core Team, which is subscribed to DOLFIN.
> https://bugs.launchpad.net/bugs/1016540
>
> Title:
>  Build problems with MPICH2 (and similar) MPI libraries
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/dolfin/+bug/1016540/+subscriptions

[Jens (jens.timmerman)]

I agree with Garth,
Using the supplied patch I am able to compile the file, but the linking then fails:

Run Build Command:/usr/bin/gmake "cmTryCompileExec/fast"
/usr/bin/gmake -f CMakeFiles/cmTryCompileExec.dir/build.make CMakeFiles/cmTryCompileExec.dir/build
gmake[1]: Entering directory `/var/tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/CMakeTmp'
/vscmnt/gent_gulpin/_apps_gent/gengar/harpertown/software/CMake/2.8.4-ictce-4.0.6/bin/cmake -E cmake_progress_report /var/tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/CMakeTmp/CMakeFiles 1
Building CXX object CMakeFiles/cmTryCompileExec.dir/scotch_test_lib.cpp.o
/apps/gent/gengar/harpertown/software/impi/4.0.2.003/bin64/mpiicpc -DMPICH_IGNORE_CXX_SEEK -xHOST -O2 -I/apps/gent/gengar/harpertown/software/SCOTCH/5.1.12b_esmumps-ictce-4.0.6/include -I/apps/gent/gengar/harpertown/software/impi/4.0.2.003/include64 -o CMakeFiles/cmTryCompileExec.dir/scotch_test_lib.cpp.o -c /var/tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/scotch_test_lib.cpp
Linking CXX executable cmTryCompileExec
/vscmnt/gent_gulpin/_apps_gent/gengar/harpertown/software/CMake/2.8.4-ictce-4.0.6/bin/cmake -E cmake_link_script CMakeFiles/cmTryCompileExec.dir/link.txt --verbose=1
/apps/gent/gengar/harpertown/software/impi/4.0.2.003/bin64/mpiicpc -xHOST -O2 -L/apps/gent/gengar/harpertown/software/icc/2011.6.233/compiler/lib/intel64 -L/apps/gent/gengar/harpertown/software/ifort/2011.6.233/compiler/lib/intel64 -L/apps/gent/gengar/harpertown/software/imkl/10.3.6.233/mkl/lib/intel64 -L/apps/gent/gengar/harpertown/software/Python/2.7.3-ictce-4.0.6/lib -L/apps/gent/gengar/harpertown/software/Boost/1.49.0-ictce-4.0.6-Python-2.7.3/lib -L/apps/gent/gengar/harpertown/software/UFC/2.0.5-ictce-4.0.6-Python-2.7.3/lib -L/apps/gent/gengar/harpertown/software/FFC/1.0.0-ictce-4.0.6-Python-2.7.3/lib -L/apps/gent/gengar/harpertown/software/FIAT/1.0.0-ictce-4.0.6-Python-2.7.3/lib -L/apps/gent/gengar/harpertown/software/Instant/1.0.0-ictce-4.0.6-Python-2.7.3/lib -L/apps/gent/gengar/harpertown/software/Viper/1.0.0-ictce-4.0.6-Python-2.7.3/lib -L/apps/gent/gengar/harpertown/software/UFL/1.0.0-ictce-4.0.6-Python-2.7.3/lib -L/apps/gent/gengar/harpertown/software/SCOTCH/5.1.12b_esmumps-ictce-4.0.6/lib -L/apps/gent/gengar/harpertown/software/Armadillo/2.4.4-ictce-4.0.6-Python-2.7.3/lib -L/apps/gent/gengar/harpertown/software/ParMETIS/4.0.2-ictce-4.0.6/lib -L/apps/gent/gengar/harpertown/software/CGAL/4.0-ictce-4.0.6-Python-2.7.3/lib -L/apps/gent/gengar/harpertown/software/zlib/1.2.7-ictce-4.0.6/lib CMakeFiles/cmTryCompileExec.dir/scotch_test_lib.cpp.o -o cmTryCompileExec -rdynamic -Wl,-Bstatic -lptscotch -lptscotcherr -Wl,-Bdynamic /apps/gent/gengar/harpertown/software/impi/4.0.2.003/lib64/libmpi.so -Wl,-rpath,/apps/gent/gengar/harpertown/software/impi/4.0.2.003/lib64
gmake[1]: Leaving directory `/var/tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/CMakeFiles/CMakeTmp'

-- [ /var/tmp/easy/DOLFIN/1.0.0/ictce-4.0.6-Python-2.7.3/ictce-4.0.6-Python-2.7.3/dolfin-1.0.0/c...

[Jens (jens.timmerman)]

I am also seeing failed test runs for cholmod and umfpack, but these don't seem to have a UMFPACK_DEBUG:BOOL=ON option?

[Garth Wells (garth-wells)]

Could you try the dev version (lp:dolfin) now? I've just pushed some changes for the build tests.

[Kenneth Hoste (kenneth-hoste)]

I've been working on this together with Jens.

We've managed to fix the failed CHOLMOD and UMFPACK test runs by checking the CMakeFiles/CMakeError.log log file, which mentioned not being able to find the UFconfig.h include file.

This could be fixed by supplying the include directory where UFconfig.h is located on the cmake command line:

cmake <other options> -DUMFPACK_INCLUDE_DIRS="/path/to/SuiteSparse/UMFPACK/include;/path/to/SuiteSparse/UFconfig" -DCHOLMOD_INCLUDE_DIRS="/path/to/SuiteSparse/CHOLMOD/include;/path/to/SuiteSparse/UFconfig"

Maybe we ran into this because we're using SuiteSparse, which we installed in a non-default path, to resolve the CHOLMOD and UNFPACK dependencies.

We haven't tested with the development version of DOLFIN, and we won't for now. Getting DOLFIN v1.0.0 to build was already quite a challenge.

To put things in perspective: we've added support for building DOLFIN and all its dependencies with a single command in EasyBuild (see github.com/hpcugent/easybuild). The DOLFIN support will be released as part of v0.9, but is already available in the develop branch of EasyBuild now. This support allows us to build DOLFIN both with a GCC and Intel-based compiler toolkit.

Maybe it's worth for you guys to look into EasyBuild, and mention it to DOLFIN users as a very easy way to build DOLFIN without much hassle. Given the dependency graph of DOLFIN (see http://users.ugent.be/~kehoste/dolfin.png), this is not exactly easy...

[Garth Wells (garth-wells)]

MPI fix committed to lp:dolfin. Tested with Intel MPI compiler, which is apparently based on MPICH.