### How to roughly estimate memory required for PETSc CG solver with petsc_amg preconditioner

253

views

0

I am trying to run simple unit cube linear elasticity problem with petsc CG solver with amg preconditioner and I suspect that it is using more than expected memory. I am not able to run a 300x300x300 problem (~81M dof) in one cluster node with 128GB of RAM. It just throws error of Bad termination. I am not posting full error description as I am more interested to know how can I estimate the memory requirement by hand so that I can verify with petsc log summary. As a simple thumb rule I expect the CG solver to use 1GB of memory for a simple problem of 1M dof and 10GB for same problem with direct sparse solver.

I have some idea of calculating this based on number of non zeros in assembled matrix and 8 byte for double precision number. But not sure if it's only applicable for sparse direct solver as I heard that PCG solver only needs to store data to do matrix-vector product in element level rather than storing full assembled matrix info on CSR format. And also how much overhead is imposed due to preconditioners, specially petsc amg and how can I estimate it.

So I would like to know if somebody knows how we can estimate memory needed for PCG solver with and without preconditioners and also some way to estimate memory needed separately for petsc amg and some other preconditioner.

I have some idea of calculating this based on number of non zeros in assembled matrix and 8 byte for double precision number. But not sure if it's only applicable for sparse direct solver as I heard that PCG solver only needs to store data to do matrix-vector product in element level rather than storing full assembled matrix info on CSR format. And also how much overhead is imposed due to preconditioners, specially petsc amg and how can I estimate it.

So I would like to know if somebody knows how we can estimate memory needed for PCG solver with and without preconditioners and also some way to estimate memory needed separately for petsc amg and some other preconditioner.

Community: FEniCS Project

### 1 Answer

4

It is difficult to estimate the memory usage for algebraic multigrid. Much of the memory will be used in the preconditioner construction and not in the CG solver (CG is the matrix operator and just a few vectors).

The memory usage will be very close to linear in the number of dofs, so you can extrapolate memory usage from smaller problems to get a good estimate. Be sure to sample during construction of the preconditioner. The memory required can be reduced by coarsening faster.

From my experience, 81M dofs will need more than 128GB.

The memory usage will be very close to linear in the number of dofs, so you can extrapolate memory usage from smaller problems to get a good estimate. Be sure to sample during construction of the preconditioner. The memory required can be reduced by coarsening faster.

From my experience, 81M dofs will need more than 128GB.

Thanks for explanation Garth. That means if I do not use any preconditioner the memory need should drastically reduced, even it should remain almost constant if I increase the number of dof, since CG only need element level information to be stored. Is it correct? But I saw somewhere in Fenics F&Q that with AMG preconditioner we have only small amount of memory overhead. Not sure if I got it right. And as you mentioned 300x300x300 problem would need more than 128GB of RAM, I would like to give one more scenario. I even tried 500x500x500 problem with 64 nodes, and got the same kind of bad termination. Is this case also questionable with these huge amount of memory(64x128GB).

written
9 months ago by
sandeep shrivastava

1

Yes, if you use no preconditioner the memory usage will decrease substantially compared to the AMG preconditioned case.

AMG has quite a high memory cost, but it is memory scalable (unlike LU). Start with small problems to understand the memory usage as a function of problem size. Since the memory usage is O(n), you will then be able to get a good a priori estimate of the required memory for a problem.

AMG has quite a high memory cost, but it is memory scalable (unlike LU). Start with small problems to understand the memory usage as a function of problem size. Since the memory usage is O(n), you will then be able to get a good a priori estimate of the required memory for a problem.

written
9 months ago by
Garth Wells

Thanks Garth. I'll give it a try. Now I think I am in a good shape of understanding memory uses for CG solver and preconditioners.

written
9 months ago by
sandeep shrivastava

Please login to add an answer/comment or follow this question.

So, run several sets of problems and track the memory usage for each of your Krylov methods, then extrapolate at a given amount of memory for the maximum number of dofs you can use for any given computer -- or partition on a compute node -- for each solver method or preconditioner choice.