Ghost cells of a Rectangular Mesh using MPI
I am trying to use MPI to speedup a diffusion equation solution, where I need to be able to read/write each mesh vertex in each time-step (for example, to assert a Dirac-delta concentration at varying nodes). When using MPI, a simple rectangular mesh will be divided (automatically) by dolfin to the different processes and it is straightforward to read in each process which nodes belong to that process using (in C++):
//To enable ghost mode: dolfin::parameters["ghost_mode"] = "shared_facet"; ... mesh = std::make_shared<dolfin::RectangleMesh>(*p0, *p1, nodesW, nodesH, "right"); mesh->str(true);
As an example output using a square mexsh with 5x5 nodes, this generates 6x6 vertices, and running on 2 MPI nodes this is the assignment to each process (I format the 6x6 grid output with vertices numbered bottom-left to top-right, with an 'X' meaning the vertex is not included in the process):
$ mpirun -n 2 ./hsl.out ... HSL MPI::rank(): 0 X X X 33 34 35 X X 26 27 28 29 X X 20 21 22 23 X X 14 15 16 17 X 7 8 9 10 11 X 1 2 3 4 5 #Nodes: 25 HSL MPI::rank(): 1 30 31 32 33 34 X 24 25 26 27 28 X 18 19 20 21 X X 12 13 14 15 X X 6 7 8 9 X X 0 1 2 X X X #Nodes: 25
There are obviously ghost-cells included in each process, but I cannot figure out a general rule as to which process owns which vertex in every case (where in the other process it would be a ghost cell). For example, it is unclear who owns #8 or #27 in the above, as each has a full complement of vertex connectivity for "right" FE triangles. (Other boundary cells are easily discernible, for example, #s: 1,7,14,20,26,33 are owned by process 1, and ghost cells in process 0).
As a general question, how should I write to the vertices (of course I must translate vertices into dof's using vertex_to_dof_map() in each process when writing), when using MPI? Should all processes write each ghost node (this would require that I separately ghost-transfer my concentration solution) or is there a way to discover in which process a vertex is "owned", and in which it is a ghost cell? Fenics clearly transfers the ghost-cells "under the hood" at some point, so I prefer to not have to do this twice. A related question is whether ghost-transfer is done both before and after calling the solver.solve() function?
Thanks for any help!