Ghost cells of a Rectangular Mesh using MPI

5 months ago by
Fenics experts:

I am trying to use MPI to speedup a diffusion equation solution, where I need to be able to read/write each mesh vertex in each time-step (for example, to assert a Dirac-delta concentration at  varying nodes).  When using MPI, a simple rectangular mesh will be divided (automatically) by dolfin to the different processes and it is straightforward to read in each process which nodes belong to that process using (in C++):

//To enable ghost mode:
dolfin::parameters["ghost_mode"] = "shared_facet";
mesh = std::make_shared<dolfin::RectangleMesh>(*p0, *p1, nodesW, nodesH, "right");

As an example output using a square mexsh with 5x5 nodes, this generates 6x6 vertices, and running on 2 MPI nodes this is the assignment to each process (I format the 6x6  grid output with vertices numbered bottom-left to top-right, with an 'X' meaning the vertex is not included in the process):

$ mpirun -n 2 ./hsl.out
HSL MPI::rank(): 0
X X X  33 34 35 
X X 26 27 28 29 
X X 20 21 22 23 
X X 14 15 16 17 
X 7 8  9  10 11 
X 1 2  3  4  5 
#Nodes: 25

HSL MPI::rank(): 1
30 31 32 33 34 X 
24 25 26 27 28 X 
18 19 20 21 X  X 
12 13 14 15 X  X 
6  7  8  9  X  X 
0  1  2  X  X  X 
#Nodes: 25

There are obviously ghost-cells included in each process, but I cannot figure out a general rule as to which process owns which vertex in every case (where in the other process it would be a ghost cell).   For example, it is unclear who owns #8 or #27 in the above, as each has a full complement of vertex connectivity for "right" FE triangles.   (Other boundary cells are easily discernible, for example, #s: 1,7,14,20,26,33  are owned by process 1, and ghost cells in process 0).

As a general question, how should I write to the vertices (of course I must translate vertices into dof's using vertex_to_dof_map() in each process when writing), when using MPI?  Should all processes write each ghost node (this would require that I separately ghost-transfer my concentration solution) or is there a way to discover in which process a vertex is "owned", and in which it is a ghost cell?  Fenics clearly transfers the ghost-cells "under the hood" at some point, so I prefer to not have to do this twice.  A related question is whether ghost-transfer is done both before and after calling the solver.solve() function?

Thanks for any help!

Community: FEniCS Project
Please login to add an answer/comment or follow this question.

Similar posts:
Search »