How to correctly set ghost mode for parallel assembly?

490
views
2
11 months ago by
I recently encountered a bit unusual behaviour when assembling forms with facet contributions coming from discontinuous Galerkin discretizations. The following example code crashes with quite a surprising error message.
from dolfin import *

mesh = UnitSquareMesh(10, 10)

BDM = FiniteElement('BDM', mesh.ufl_cell(), 1)
W = FunctionSpace(mesh, BDM)
u = TrialFunction(W)
w = TestFunction(W)

a = inner(u, w)*dx
a_broken = inner(avg(u), avg(w))*dS

a = a + a_broken

parameters['ghost_mode'] = 'shared_facet'
print(parameters['ghost_mode'])  # prints shared_facet
A = assemble(a)  # fails in parallel run
print(A)​
This fails when run in parallel on the assembly with error:
*** Error:   Unable to assemble form.
*** Reason:  Incorrect mesh ghost mode "none" (expected "shared_vertex" or "shared_facet" for interior facet integrals in parallel).
*** Where:   This error was encountered inside AssemblerBase.cpp.
Please note that the error message claims that the reason for this error is wrongly set ghost mode. But the ghost mode is set before calling assemble. The same result can be obtained by setting ghost mode to shared_vertex. I see this using Fenics from Ubuntu ppa as well as Fenics 2017.1 compiled from source. Is there some way how to set everything properly for parallel run?
Community: FEniCS Project

4
11 months ago by
The parameter ghost_mode controls the way in which the mesh is being partitioned, and therefore you have to set this parameter before the mesh definition. Check this code snippet for clarification (comment and uncomment the first line and see how the output changes):

parameters['ghost_mode'] = 'shared_facet'
mesh = UnitSquareMesh(10, 10)
print mesh.ghost_mode()​

So, the correct way to write your code should be:

from dolfin import *

parameters['ghost_mode'] = 'shared_facet'

mesh = UnitSquareMesh(10, 10)

BDM = FiniteElement('BDM', mesh.ufl_cell(), 1)
W = FunctionSpace(mesh, BDM)
u = TrialFunction(W)
w = TestFunction(W)

a = inner(u, w)*dx
a_broken = inner(avg(u), avg(w))*dS

a = a + a_broken

A = assemble(a)
print(A)​​
Thanks! Is there some document where I could find information like this on my own?
written 11 months ago by TomL
This is not very well documented. The example appears in documentation of Biharmonic demo http://fenics.readthedocs.io/projects/dolfin/en/2017.1.0.post0/demos/biharmonic/python/demo_biharmonic.py.html.
written 11 months ago by Jan Blechta
Ghosted mesh support is still experimental and the interface for setting it is likely to change, which is a reason it is not yet documented.
written 11 months ago by Garth Wells