Navier-Stokes demo does not work in parallel

[Mikael Mortensen]

The Navier-Stokes demo works fine with one processor but fails when I'm trying two. I get the error message

[mmo@mk083:python] mpirun -np 2 python demo_navier-stokes.py
Process 0: Partitioned mesh, edge cut is 8.
Process 1: Partitioned mesh, edge cut is 8.
[mk083:02497] *** Process received signal ***
[mk083:02497] Signal: Segmentation fault (11)
[mk083:02497] Signal code: Address not mapped (1)
[mk083:02497] Failing at address: 0x4
[mk083:02498] *** Process received signal ***
[mk083:02498] Signal: Segmentation fault (11)
[mk083:02498] Signal code: Address not mapped (1)
[mk083:02498] Failing at address: 0x4
[mk083:02498] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) [0x7f19e769ac60]
[mk083:02498] [ 1] /home/mmo/Fenics/dorsal/test/distance-to-boundary/dolfin/libdolfin.so.0.9(_ZN6dolfin16MeshPartitioning15number_entitiesERKNS_4MeshEj+0xa0f) [0x7f19e4d7bfcf]
[mk083:02498] [ 2] /home/mmo/Fenics/dorsal/test/distance-to-boundary/dolfin/libdolfin.so.0.9(_ZN6dolfin6DofMapC1EN5boost10shared_ptrIKN3ufc6dofmapEEERNS_4MeshE+0x3f2) [0x7f19e4bba162]
[mk083:02498] [ 3] /home/mmo/Fenics/dorsal/test/lib/python2.7/dist-packages/dolfin/_cpp.so(+0x151567) [0x7f19e5220567]
[mk083:02498] [ 4] python(PyEval_EvalFrameEx+0x42be) [0x49a54e]
[mk083:02498] [ 5] python(PyEval_EvalCodeEx+0x145) [0x49d325]
[mk083:02498] [ 6] python() [0x4c4526]
[mk083:02498] [ 7] python(PyObject_Call+0x44) [0x45d864]
[mk083:02498] [ 8] python() [0x45f43f]
[mk083:02498] [ 9] python(PyObject_Call+0x44) [0x45d864]
[mk083:02498] [10] python() [0x48ae3f]
[mk083:02498] [11] python() [0x48928d]
[mk083:02497] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfc60) [0x7f83593f7c60]
[mk083:02497] [ 1] /home/mmo/Fenics/dorsal/test/distance-to-boundary/dolfin/libdolfin.so.0.9(_ZN6dolfin16MeshPartitioning15number_entitiesERKNS_4MeshEj+0xa0f) [0x7f8356ad8fcf]
[mk083:02497] [ 2] /home/mmo/Fenics/dorsal/test/distance-to-boundary/dolfin/libdolfin.so.0.9(_ZN6dolfin6DofMapC1EN5boost10shared_ptrIKN3ufc6dofmapEEERNS_4MeshE+0x3f2) [0x7f8356917162]
[mk083:02497] [ 3] /home/mmo/Fenics/dorsal/test/lib/python2.7/dist-packages/dolfin/_cpp.so(+0x151567) [0x7f8356f7d567]
[mk083:02497] [ 4] python(PyEval_EvalFrameEx+0x42be) [0x49a54e]
[mk083:02497] [ 5] python(PyEval_EvalCodeEx+0x145) [0x49d325]
[mk083:02497] [ 6] python() [0x4c4526]
[mk083:02498] [12] python(PyObject_Call+0x44) [0x45d864]
[mk083:02498] [13] python(PyEval_EvalFrameEx+0x9be) [0x496c4e]
[mk083:02498] [14] python(PyEval_EvalCodeEx+0x145) [0x49d325]
[mk083:02498] [15] python() [0x4c4526]
[mk083:02498] [16] python(PyObject_Call+0x44) [0x45d864]
[mk083:02498] [17] python() [0x45f43f]
[mk083:02498] [18] python(PyObject_Call+0x44) [0x45d864]
[mk083:02498] [19] python(PyEval_EvalFrameEx+0x9be) [0x496c4e]
[mk083:02498] [20] python(PyEval_EvalCodeEx+0x145) [0x49d325]
[mk083:02498] [21] python() [0x4c4526]
[mk083:02498] [22] python(PyObject_Call+0x44) [0x45d864]
[mk083:02498] [23] python() [0x45f43f]
[mk083:02498] [24] python(PyObject_Call+0x44) [0x45d864]
[mk083:02498] [25] python() [0x48ae3f]
[mk083:02498] [26] python() [0x48928d]
[mk083:02498] [27] python(PyObject_Call+0x44) [0x45d864]
[mk083:02498] [28] python(PyEval_EvalFrameEx+0x9be) [0x496c4e]
[mk083:02498] [29] python(PyEval_EvalCodeEx+0x145) [0x49d325]
[mk083:02498] *** End of error message ***
[mk083:02497] [ 7] python(PyObject_Call+0x44) [0x45d864]
[mk083:02497] [ 8] python() [0x45f43f]
[mk083:02497] [ 9] python(PyObject_Call+0x44) [0x45d864]
[mk083:02497] [10] python() [0x48ae3f]
[mk083:02497] [11] python() [0x48928d]
[mk083:02497] [12] python(PyObject_Call+0x44) [0x45d864]
[mk083:02497] [13] python(PyEval_EvalFrameEx+0x9be) [0x496c4e]
[mk083:02497] [14] python(PyEval_EvalCodeEx+0x145) [0x49d325]
[mk083:02497] [15] python() [0x4c4526]
[mk083:02497] [16] python(PyObject_Call+0x44) [0x45d864]
[mk083:02497] [17] python() [0x45f43f]
[mk083:02497] [18] python(PyObject_Call+0x44) [0x45d864]
[mk083:02497] [19] python(PyEval_EvalFrameEx+0x9be) [0x496c4e]
[mk083:02497] [20] python(PyEval_EvalCodeEx+0x145) [0x49d325]
[mk083:02497] [21] python() [0x4c4526]
[mk083:02497] [22] python(PyObject_Call+0x44) [0x45d864]
[mk083:02497] [23] python() [0x45f43f]
[mk083:02497] [24] python(PyObject_Call+0x44) [0x45d864]
[mk083:02497] [25] python() [0x48ae3f]
[mk083:02497] [26] python() [0x48928d]
[mk083:02497] [27] python(PyObject_Call+0x44) [0x45d864]
[mk083:02497] [28] python(PyEval_EvalFrameEx+0x9be) [0x496c4e]
[mk083:02497] [29] python(PyEval_EvalCodeEx+0x145) [0x49d325]
[mk083:02497] *** End of error message ***

I use dolfin 0.9.11+.

Mikael

[Johannes Ring (johannr)]

I see the same error in 1.0-beta.

Also the C++ version of this demo fails:

johannr@simula-x61:cpp$ mpirun -np 2 ./demo_navier-stokes
Process 0: Partitioned mesh, edge cut is 8.
Process 1: Partitioned mesh, edge cut is 8.
demo_navier-stokes: /build/buildd/dolfin-1.0-beta/dolfin/mesh/MeshTopology.h:64: dolfin::uint dolfin::MeshTopology::size(dolfin::uint) const: Assertion `num_entities' failed.
[simula-x61:11097] *** Process received signal ***
[simula-x61:11097] Signal: Aborted (6)
[simula-x61:11097] Signal code: (-6)
demo_navier-stokes: /build/buildd/dolfin-1.0-beta/dolfin/mesh/MeshTopology.h:64: dolfin::uint dolfin::MeshTopology::size(dolfin::uint) const: Assertion `num_entities' failed.
[simula-x61:11096] *** Process received signal ***
[simula-x61:11096] Signal: Aborted (6)
[simula-x61:11096] Signal code: (-6)
[simula-x61:11096] [ 0] [0x5f9410]
[simula-x61:11096] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0x4b25a82]
[simula-x61:11096] [ 2] /lib/tls/i686/cmov/libc.so.6(__assert_fail+0xf8) [0x4b1b718]
[simula-x61:11096] [ 3] /usr/lib/libdolfin.so.1.0(_ZNK6dolfin4Mesh4initEj+0xf4) [0x434654]
[simula-x61:11096] [ 4] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12MeshFunctionIjE4initEj+0x37) [0x215877]
[simula-x61:11096] [ 5] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12MeshFunctionIjEC1ERKNS_4MeshEj+0xe6) [0x215d66]
[simula-x61:11096] [ 6] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12ParallelData21global_entity_indicesEj+0xe2) [0x40b702]
[simula-x61:11096] [ 7] /usr/lib/libdolfin.so.1.0(_ZN6dolfin16MeshPartitioning15number_entitiesERKNS_4MeshEj+0x19a) [0x40772a]
[simula-x61:11096] [ 8] /usr/lib/libdolfin.so.1.0(_ZN6dolfin6DofMapC1EN5boost10shared_ptrIKN3ufc6dofmapEEERNS_4MeshE+0x217) [0x265f27]
[simula-x61:11096] [ 9] ./demo_navier-stokes(_ZN14VelocityUpdate22Form_0_FunctionSpace_0C1ERN6dolfin4MeshE+0xa2) [0x80ab1a2]
[simula-x61:11096] [10] ./demo_navier-stokes(main+0x9f) [0x807325f]
[simula-x61:11096] [11] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0x4b0ebd6]
[simula-x61:11096] [12] ./demo_navier-stokes() [0x80729e1]
[simula-x61:11096] *** End of error message ***
[simula-x61:11097] [ 0] [0x706410]
[simula-x61:11097] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0x773aa82]
[simula-x61:11097] [ 2] /lib/tls/i686/cmov/libc.so.6(__assert_fail+0xf8) [0x7730718]
[simula-x61:11097] [ 3] /usr/lib/libdolfin.so.1.0(_ZNK6dolfin4Mesh4initEj+0xf4) [0x434654]
[simula-x61:11097] [ 4] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12MeshFunctionIjE4initEj+0x37) [0x215877]
[simula-x61:11097] [ 5] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12MeshFunctionIjEC1ERKNS_4MeshEj+0xe6) [0x215d66]
[simula-x61:11097] [ 6] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12ParallelData21global_entity_indicesEj+0xe2) [0x40b702]
[simula-x61:11097] [ 7] /usr/lib/libdolfin.so.1.0(_ZN6dolfin16MeshPartitioning15number_entitiesERKNS_4MeshEj+0x19a) [0x40772a]
[simula-x61:11097] [ 8] /usr/lib/libdolfin.so.1.0(_ZN6dolfin6DofMapC1EN5boost10shared_ptrIKN3ufc6dofmapEEERNS_4MeshE+0x217) [0x265f27]
[simula-x61:11097] [ 9] ./demo_navier-stokes(_ZN14VelocityUpdate22Form_0_FunctionSpace_0C1ERN6dolfin4MeshE+0xa2) [0x80ab1a2]
[simula-x61:11097] [10] ./demo_navier-stokes(main+0x9f) [0x807325f]
[simula-x61:11097] [11] /lib...

[Garth Wells (garth-wells)]

We really need to have some of the demos running in parallel as part of
the test suite.

Garth

On 22/08/11 09:46, Johannes Ring wrote:
> I see the same error in 1.0-beta.
>
> Also the C++ version of this demo fails:
>
> johannr@simula-x61:cpp$ mpirun -np 2 ./demo_navier-stokes
> Process 0: Partitioned mesh, edge cut is 8.
> Process 1: Partitioned mesh, edge cut is 8.
> demo_navier-stokes: /build/buildd/dolfin-1.0-beta/dolfin/mesh/MeshTopology.h:64: dolfin::uint dolfin::MeshTopology::size(dolfin::uint) const: Assertion `num_entities' failed.
> [simula-x61:11097] *** Process received signal ***
> [simula-x61:11097] Signal: Aborted (6)
> [simula-x61:11097] Signal code: (-6)
> demo_navier-stokes: /build/buildd/dolfin-1.0-beta/dolfin/mesh/MeshTopology.h:64: dolfin::uint dolfin::MeshTopology::size(dolfin::uint) const: Assertion `num_entities' failed.
> [simula-x61:11096] *** Process received signal ***
> [simula-x61:11096] Signal: Aborted (6)
> [simula-x61:11096] Signal code: (-6)
> [simula-x61:11096] [ 0] [0x5f9410]
> [simula-x61:11096] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0x4b25a82]
> [simula-x61:11096] [ 2] /lib/tls/i686/cmov/libc.so.6(__assert_fail+0xf8) [0x4b1b718]
> [simula-x61:11096] [ 3] /usr/lib/libdolfin.so.1.0(_ZNK6dolfin4Mesh4initEj+0xf4) [0x434654]
> [simula-x61:11096] [ 4] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12MeshFunctionIjE4initEj+0x37) [0x215877]
> [simula-x61:11096] [ 5] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12MeshFunctionIjEC1ERKNS_4MeshEj+0xe6) [0x215d66]
> [simula-x61:11096] [ 6] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12ParallelData21global_entity_indicesEj+0xe2) [0x40b702]
> [simula-x61:11096] [ 7] /usr/lib/libdolfin.so.1.0(_ZN6dolfin16MeshPartitioning15number_entitiesERKNS_4MeshEj+0x19a) [0x40772a]
> [simula-x61:11096] [ 8] /usr/lib/libdolfin.so.1.0(_ZN6dolfin6DofMapC1EN5boost10shared_ptrIKN3ufc6dofmapEEERNS_4MeshE+0x217) [0x265f27]
> [simula-x61:11096] [ 9] ./demo_navier-stokes(_ZN14VelocityUpdate22Form_0_FunctionSpace_0C1ERN6dolfin4MeshE+0xa2) [0x80ab1a2]
> [simula-x61:11096] [10] ./demo_navier-stokes(main+0x9f) [0x807325f]
> [simula-x61:11096] [11] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0x4b0ebd6]
> [simula-x61:11096] [12] ./demo_navier-stokes() [0x80729e1]
> [simula-x61:11096] *** End of error message ***
> [simula-x61:11097] [ 0] [0x706410]
> [simula-x61:11097] [ 1] /lib/tls/i686/cmov/libc.so.6(abort+0x182) [0x773aa82]
> [simula-x61:11097] [ 2] /lib/tls/i686/cmov/libc.so.6(__assert_fail+0xf8) [0x7730718]
> [simula-x61:11097] [ 3] /usr/lib/libdolfin.so.1.0(_ZNK6dolfin4Mesh4initEj+0xf4) [0x434654]
> [simula-x61:11097] [ 4] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12MeshFunctionIjE4initEj+0x37) [0x215877]
> [simula-x61:11097] [ 5] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12MeshFunctionIjEC1ERKNS_4MeshEj+0xe6) [0x215d66]
> [simula-x61:11097] [ 6] /usr/lib/libdolfin.so.1.0(_ZN6dolfin12ParallelData21global_entity_indicesEj+0xe2) [0x40b702]
> [simula-x61:11097] [ 7] /usr/lib/libdolfin.so.1.0(_ZN6dolfin16MeshPartitioning15number_entitiesERKNS_4MeshEj+0x19a) [0x40772a]
> [simula-x61:11097] [ 8] /usr/lib/libdolfin.so.1.0(_ZN6dolfin6DofMapC1EN5boost10shared_ptrIKN3ufc6dofmapEEERNS_4MeshE+0x217) [0x265f27]
> [simula...

[Garth Wells (garth-wells)]

First problem is that the demo make a call to

  mesh = refine(mesh)

which doesn't not work in parallel. I will add an error to dolfin::refine.

Second problem looks to be that the demo uses ILU preconditioning. This can be problematic with the zero pressure block in the matrix. There are some options to use a non-zero shift in the preconditioner.

[Garth Wells (garth-wells)]

Turns out that PETSc does not have its own parallel ILU preconditioner.

A 'fix' (not_working_in_parallel()) has been added to the refinement functions.

[Anders Logg (logg)]

On Mon, Aug 22, 2011 at 09:46:23PM -0000, Garth Wells wrote:
> First problem is that the demo make a call to
>
> mesh = refine(mesh)
>
> which doesn't not work in parallel. I will add an error to
> dolfin::refine.
>
> Second problem looks to be that the demo uses ILU preconditioning. This
> can be problematic with the zero pressure block in the matrix. There are
> some options to use a non-zero shift in the preconditioner.

There's really no reason for using refine in that demo. I'll replace
the mesh so we can get the demo to run in parallel. It would be really
good if we could demonstrate some good speedup, for this and other
demos, but this one in particular.

Parallel refinement is targeted for 1.1.

--
Anders