The ABINIT Users Mailing List ( CLOSED )

[Francois Bottin <Francois.Bottin@cea.fr>]

Dear Dr. Dumont,
Concerning your problems with ncache, I'm agree with Pierre-Matthieu. It 
is needed to obtain (by hand) the optimal value of ncache in the FFT 
routines.

Guillaume Dumont wrote:

> Dear Dr Bottin,
>
> I tried to reproduce your superlinear scaling up to 144 cpus. Here are 
> the results. The scaling is superlinear up to 54 cpus for your gold 
> case. However, keeping the number of processors constant, some sets of 
> npband and npfft do not give the superlinear behavior (see graph 
> speedup.eps.)

It is normal. The worse scaling will be obtained for the limit cases: 
(npband=nproc,npfft=1) or (npfft=nproc,npband=1). In this respect, the 
superlinear scaling is obtain for an optimal distribution (npband;npfft) 
, for each number of processors nprocs.  For example, if nprocs=108, the 
two better distributions obtained are (npfft=3, npband=36) and 
(npfft=4,npband=27), over 10 distributions.

> For the superlinear regime most of the time is spent in the lobpcgxx 
> routine, but as the number of processors increase more and more time 
> is spent in gstate->kpgsph.

I don't find that.

> I also noticed that the memory requirement is proportional to the 
> number of processors ( memory.eps). This is causing problems with 
> cases where you need more than the memory accessible to a single 
> processor. For example, I tried to run a total energy calculation on a 
> 216 atoms GaAsN supercell with nband 480 and ngfft 180 180 180. I was 
> able to run it on 32 processors and it did 2 scf cycles and then 
> crashed with an error message indicating that the memory need exceeded 
> the available memory.
>
> Is there a reason why the code as such a great memory need? Why did 
> the code run for 2 scf cycles and than crashed? Shouldn't it allocate 
> all the memory before doing the calculation? (Memory leeks?)

All the memory is not allocated before doing the calculation. Some 
arrays, allocated in prep_*, are used to perform the alltoall 
(transpostion) within one row or one column of the 2D processor grid. 
Perhaps these ones become too large. We will check this point. Does this 
problem appear for all the distributions (npband;npfft)? Does this 
problem exist before the 2 SCF? Is there any swap in your calculations 
of speedup?

> This calculation needs a little more than 4 GB on a single processor run.
>
> To answer your other questions:
>
> In the cases of both Au and GaAsN systems? For gold, the code is two
> times faster (if I remember correctly) with the -O3 flag compilation.
>
> I did not test the gold on case with the -O2 flag, but I'll let you 
> know when I do it.
>
> Does the lobpcg part in these two systems weight equally? In Au, the
> lobpcg part corresponds approximatively to the total time. Its perfect
> scaling gives the supelinear behaviour of ABINIT.
>
> Does your FFT part (fourwf) strongly increase (more than 2 times)
> between 1 and 32 processors? And what is its weight? Even if this FFT is
> strongly optimized, the scaling does not remain linear.
>
> Unfortunately some of the calculations where done with timopt 2 
> instead of -1 or -2 so I cannot answer this question yet.
>
> Regards,
>
> --
> Guillaume Dumont
> =========================
> guillaume.dumont.1@umontreal.ca <mailto:guillaume.dumont.1@umontreal.ca>
> dumont.guillaume@gmail.com <mailto:dumont.guillaume@gmail.com>
> (514) 341 5298
> (514) 343 6111 ext. 13279 

Best regards,
Francois Bottin

--
##############################################################
Francois Bottin                    tel: 01 69 26 41 73
CEA/DIF                            fax: 01 69 26 70 77
BP 12 Bruyeres-le-Chatel         email: Francois.Bottin@cea.fr
##############################################################