### Running multiple solve in parallel using MPI

418
views
5
6 months ago by
Hello, I'm trying to solve a PDE for different parameters in parallel using MPI.
After some support ( https://fenicsproject.org/qa/3772/spawning-independent-dolfin-processes-with-mpi4py/ ), I was able to use the following code to force each process to operate only on the local problem (and not partition the mesh).

import dolfin as do
from petsc4py import PETSc
from mpi4py import MPI
comm = PETSc.Comm(MPI.COMM_SELF)
mesh = do.UnitSquareMesh(comm,10,10)​
This solution doesn't work any more. At least in my machine I get the error:
~/.emacs.d/.python-environments/tm3/lib/python3.5/site-packages/dolfin/cpp/mesh.py in __init__(self, *args)
6385
6386         """
-> 6387         _mesh.UnitSquareMesh_swiginit(self, _mesh.new_UnitSquareMesh(*args))
6388     __swig_destroy__ = _mesh.delete_UnitSquareMesh
6389 UnitSquareMesh_swigregister = _mesh.UnitSquareMesh_swigregister

TypeError: (size_t) expected positive 'int' for argument 1

Did something change in the mesh generation lately?
Also, right now I would need to use the constructor do.Mesh and load an XML mesh instead of using do.UnitSquareMesh. I.e. I would like to do:
import dolfin as do
from petsc4py import PETSc
from mpi4py import MPI
comm = PETSc.Comm(MPI.COMM_SELF)
mesh = do.Mesh(comm,'mesh.xml')​​

Any idea? Thanks!

Community: FEniCS Project
I'm also trying to solve the same set of PDEs with varying parameters (hundreds of different cases). However, I can't get it to work for me. Could you provide a minimal example? I am using version 2017.2.0.
written 7 weeks ago by Miguel Rodriguez

1
6 months ago by
What fenics version do you have? The code worked for me with 2017.2.0 (docker image stable).
Anyway, it is better to use fenics's methods to get MPI communicator. You wont get into these type of problems then (the problem you have is discrepancy in MPI wrapped as mpi4py communicator and as petsc4py communicator).

In 2017.2.0 and earlier use dolfin.mpi_comm_self()
In 2018.1.0dev use dolfin.MPI.comm_self

These both return different python objects, but their use should be consistent throughout the fenics (also for Mesh creation).
I'm using 2017.2.0.dev0 from source. I tried the docker installation and it works. I will look into what I did wrong during the installation then.

Regarding the suggestion on the communicators, if I use the dolfin communicator to build the mesh I should use them as well for any other MPI part of my code, right? It would create problem passing a dolfin communicator to the mesh and using an mpi4py communicator elsewhere, right?
written 6 months ago by Daniele Bigoni
1
Strictly, there is no such thing as dolfin communicator. Dolfin just wraps standard MPI C library into its own C++ class Dolfin::MPI.
The problem is how the C++ code is binded by python, i.e. the SWIG or pybind11 (pybind is recent upgrade). I guess, when you build dolfin, SWIG wrappers for python are generated based on what python mpi library is available (mpi4py, petsc4py, ...). Pybind11 binds using mpi4py.

If you do not want to rebuild dolfin with mpi4py then I would suggest you to call dolfin method to get a communicator (dolfin.mpi_comm_self()). Print this thing in Python to check it is instance of petsc. It should have method tompi4py() that will return you mpi4py communicator and use that one for your custom communication.

However, there are difficulties when duplicate communicators appear, see https://bitbucket.org/mpi4py/mpi4py/issues/41/unable-to-convert-duplicated-petsc4py-comm

Stick to the rule: "In short, if you do MPI, use mpi4py as much as you can, that the "right" tool for the task."
written 6 months ago by Michal Habera