How to roughly estimate memory required for PETSc CG solver with petsc_amg preconditioner

3 months ago by
I am trying to run simple unit cube linear elasticity problem with petsc CG solver with amg preconditioner and I suspect that it is using more than expected memory. I am not able to run a 300x300x300 problem (~81M dof) in one cluster node with 128GB of RAM. It just throws error of Bad termination. I am not posting full error description as I am more interested to know how can I estimate the memory requirement by hand so that I can verify with petsc log summary. As a simple thumb rule I expect the CG solver to use 1GB of memory for a simple problem of 1M dof and 10GB for same problem with direct sparse solver.

I have some idea of calculating this based on number of non zeros in assembled matrix and 8 byte for double precision number. But not sure if it's only applicable for sparse direct solver as I heard that PCG solver only needs to store data to do matrix-vector product in element level rather than storing full assembled matrix info on CSR format. And also how much overhead is imposed due to preconditioners, specially petsc amg and how can I estimate it.

So I would like to know if somebody knows how we can estimate memory needed for PCG solver with and without preconditioners and also some way to estimate memory needed separately for petsc amg and some other preconditioner.
Community: FEniCS Project
If you want an estimate, why don't you run a few simulations and plot the memory footprint as a function of dofs?
written 3 months ago by pf4d  
Because I need a reference to compare. I want to verify what I see in simulation is reasonable or not.
written 3 months ago by sandeep shrivastava  
...not sure I follow.
written 3 months ago by pf4d  
Evan, I mean to say, how can I calculate this by some formula by hand, if something is little straight forward. But as Garth mentioned, I think it's difficult to estimate this with a straight formula.
written 3 months ago by sandeep shrivastava  
Is that a question?  Sorry, let me clarify; the best fit line for the set of memory usage in MB sampled at each number of dofs for small problems (meaning orders of magnitude less than 500 x 500 x 500) will provide you a very good estimate of the maximum number of dofs you can use for any given computer.  I originally stated this as a question because I was unsure if you wanted an analytical expression for the memory usage for some higher purpose.

So, run several sets of problems and track the memory usage for each of your Krylov methods, then extrapolate at a given amount of memory for the maximum number of dofs you can use for any given computer -- or partition on a compute node -- for each solver method or preconditioner choice.
written 3 months ago by pf4d  
Thanks Evan. You are right, I think that's the only way we can estimate some rough numbers.
written 3 months ago by sandeep shrivastava  

1 Answer

3 months ago by
It is difficult to estimate the memory usage for algebraic multigrid.  Much of the memory will be used in the preconditioner construction and not in the CG solver (CG is the matrix operator and just a few vectors).

The memory usage will be very close to linear in the number of dofs, so you can extrapolate memory usage from smaller problems to get a good estimate. Be sure to sample during construction of the preconditioner. The memory required can be reduced by coarsening faster.

From my experience,  81M dofs will need more than 128GB.
Thanks for explanation Garth. That means if I do not use any preconditioner the memory need should drastically reduced, even it should remain almost constant if I increase the number of dof, since CG only need element level information to be stored. Is it correct? But I saw somewhere in Fenics F&Q that with AMG preconditioner we have only small amount of memory overhead. Not sure if I got it right. And as you mentioned 300x300x300 problem would need more than 128GB of RAM, I would like to give one more scenario. I even tried 500x500x500 problem with 64 nodes, and got the same kind of bad termination. Is this case also questionable with these huge amount of memory(64x128GB).
written 3 months ago by sandeep shrivastava  
Yes, if you use no preconditioner the memory usage will decrease substantially compared to the AMG preconditioned case.

AMG has quite a high memory cost, but it is memory scalable (unlike LU). Start with small problems to understand the memory usage as a function of problem size. Since the memory usage is O(n), you will then be able to get a good a priori estimate of the required memory for a problem.

written 3 months ago by Garth Wells  
Thanks Garth. I'll give it a try. Now I think I am in a good shape of understanding memory uses for CG solver and preconditioners.
written 3 months ago by sandeep shrivastava  
Please login to add an answer/comment or follow this question.

Similar posts:
Search »