The ABINIT Users Mailing List ( CLOSED )

[<Marc.TORRENT@cea.fr>]

Hi David,

1) Did you try with the last corrections contained in the last 5.8.3 bzr 
revision or 5.8.4 (at least revision 507) ? Muriel Delaveau and I found 
several improvements for the writing of the WFK file with MPI-IO; these 
corrections improve the portability of the code and have been merged in the 
revision 507. We found that the code produced crashes on several 
architectures because of wrong treatment of buffers... and hope to have 
correct that.

2) If you want to be able to use the file with anaddb you have to :
- use the last 5.8.3 branch (or 5.8.4)
or
- use the --enable-mpi-io-buggy option when building the code (unuseful after 
5.8.3 rev507); but you could have buffer problems in that case.


We tested the new changes on ifort and gcc43 with mpich, open-mpi.

In band-fft, the memory is splitted for the wfk but not for other quantities, 
especially if you use PAW. We plan to correct that soon.

Marc




-------- Message d'origine--------
De: David Waroquiers [mailto:david.waroquiers@uclouvain.be]
Date: ven. 19/06/2009 12:29
À: forum@abinit.org
Objet : [abinit-forum] Band/FFT parallelism on large systems
 
Hello all,

I have tried to use the band/fft parallelism on a large supercell (a-SiO2, 72
atoms and 108 atoms). I encountered a problem while using a lot of bands (4480
bands). It reaches convergences but crashes at the end of the run when it is
supposed to write the WFK file (outwf call). I tried to run a calculation with
16, 32 and 64 processors.

I have tried with fewer bands (640) and it works.
Do you have any idea how to overcome this problem ? The WFK file is supposed 
to
be 4 GB and the available memory on the clusters is more than that. By the 
way,
in the band/fft parallelism approach, the memory for the wfk is split into the
different cpus, isn't it ?

I encountered another problem while using cut3d to analyse the wfk generated
with the band/fft parallelism. It does not recognise the file as a valid wfk
file (about the same message as when band/fft parallelism didn't allow to
restart with a different number of processors, before version 5.8 if I'm
right). Any idea too ?

My input file is hereafter and the log messages are after the input file. I'm
using public version 5.8.3, revision 485 and the machines used are the "green"
clusters in UCL :  102 Dual Quad-Core Xeon L5420/2.5GHz in Dell Blade M1000e
with 16 GB (or 32 GB for some nodes) per node of 8 processors.

Thanks a lot

David Waroquiers
PhD Student
UCL - PCPM - ETSF




My input file :

# Amorphous SiO2 : Generation of the WFK file needed for the KSS (for GW
corrections)
# Dataset 1 : GS calculation (_DEN generation)
# Dataset 2 : GS calculation with many bands (_WFK generation)

  ndtset      2
  jdtset       1 2
  timopt      2

# Dataset 1 : _DEN file generation (Density)

  tolvrs1     1.0d-12
  prtden1    1
  nstep1     5 #5 for testing
  iscf1        7
  npulayit1  7
  nband1    256

# Dataset 2 : _WFK file (Wavefunction)

  tolwfr2     1.0d-12
  nband2     4480
  nbdbuf2    384
  istwfk2      1
  iscf2         7
  nstep2      5 #5 for testing
  getden2    1

# Options for Band/FFT Parallelism

  paral_kgb   1
  wfoptalg    14
  nloalg      4
  fftalg      401
  iprcch      4
  intxc       0
  istwfk      1
  fft_opt_lob 2
  npfft       1
  npband      16 #32 #64

# K-point mesh

  kptopt      0
  kpt         0.0 0.0 0.0

# System definition
# Unit cell

  acell       1.9465690950E+01  1.9465690950E+01  1.9465690950E+01
  rprim       1 0 0
              0 1 0
              0 0 1

# Atom types

  ntypat      2
  znucl       8 14

# Atoms and coordinates

  natom       72
  typat       48*1 24*2
  xcart       1.8342971905E+01  1.0013093348E+01  4.9948115472E+00
              1.8450118788E+01  5.1100335358E+00  1.1410341879E+01
              3.0243029960E+00  1.7006888337E+01  1.0689037523E+01
              6.3068666011E+00  1.4446482399E+01  7.9505060279E+00
              1.9178811503E+01  4.4712567836E-01  3.4641995090E+00
              9.5178783093E+00  1.2762912471E+01  1.4947329016E+01
              1.7402433472E+01  4.7067303120E+00  1.5833402903E+00
              1.0623164695E+01  2.7299953166E+00  8.7471659694E+00
              1.2931573871E+01  1.8128981231E+01  6.7007362518E+00
              1.8660924236E+01  1.4792395464E+01  3.1319031106E+00
              7.0217232014E+00  6.3190579071E+00  2.1266991430E+00
              4.1181163909E-01  5.0929210080E+00  5.7193503290E+00
              7.6209880479E+00  1.5443775482E+00  6.1023412080E-01
              1.7923134211E+01  9.4056919719E+00  1.3628670860E+01
              1.4710748045E+01  9.1118601940E+00  1.7566857742E+01
              1.0411995344E+01  1.0041061607E+00  1.5870123306E+01
              1.0980496920E+01  1.3629862231E+01  6.8821852197E+00
              1.2756648650E+01  9.3922889131E+00  1.2966781879E+01
              1.3710153187E+01  2.2151381385E+00  1.9176017166E+01
              5.9015247795E+00  1.8254646045E+01  1.6364133902E+01
              3.5889689987E+00  8.6729161022E+00  4.9047876611E+00
              1.4649631278E+01  1.1782133781E+01  2.4189697381E+00
              1.3094524372E+01  1.5574388332E+01  1.1017906884E+01
              1.8122798453E+00  1.5904671691E+01  1.5390374184E+01
              8.0934509994E+00  9.9606459884E+00  5.6351418737E+00
              1.0388873243E+01  1.1258002356E+01  1.9535431306E+01
              1.7801695829E+01  1.5681701759E+01  1.1954743795E+01
              2.9395289639E+00  3.6212308778E+00  1.4808160737E+00
              1.3785141980E+01  3.1146153451E+00  4.7897808777E+00
              6.6125694236E+00  3.8955369666E+00  1.1802613942E+01
              1.0543336669E+00  8.8480531151E+00  9.2302571597E+00
              1.5034376672E+01  1.3207034271E+01  1.5126390258E+01
              1.9223920516E+01  6.5595988246E-01  1.3020475817E+01
              6.6553078921E+00  5.1934209327E+00  6.9894256581E+00
              1.4918361618E+01  3.1596212425E+00  1.4324193688E+01
              8.0804273193E+00  7.9884008127E+00  1.4307619386E+01
              5.7570753518E+00  1.3551949199E+01  1.8079850277E+01
              1.2833144388E+01  6.9576781789E+00  1.8702339976E+00
              2.7890960157E+00  1.7032376017E+01  6.7568473875E-01
              8.6760457768E+00  1.0859908527E+01  1.0407253204E+01
              6.3690907257E+00  2.2769273004E-01  8.2629069843E+00
              1.4623475391E+01  1.7952319809E+01  1.5406783784E+01
              1.5775821227E+01  1.3896960139E+01  6.9539101570E+00
              1.5477566296E+01  1.7519166868E+00  9.5117606862E+00
              3.1098755647E+00  7.2414373656E+00  1.3444441571E+01
              2.9576688783E+00  1.7045648497E+01  5.6738016905E+00
              9.7659864282E+00  1.6334927247E+01  1.8709494220E+01
              5.6015780233E+00  4.6820174692E+00  1.6849684158E+01
              1.3193293623E+01  1.6296954721E+00  7.4269549058E+00
              4.0153579861E+00  1.6089810803E+01  1.7511617105E+01
              1.5080013653E+01  1.5674902127E+01  1.3378910449E+01
              1.3468917372E+01  1.2396666756E+00  1.6246453919E+01
              3.1443097800E-01  3.4518653529E+00  3.0738155414E+00
              1.6864054813E+01  1.2620177700E+01  4.3720388857E+00
              1.3252228290E+01  1.5343974821E+01  7.9284144425E+00
              8.4872425534E+00  1.9054897865E+01  1.7815010425E+01
              1.5170448087E+01  1.0186883021E+01  1.4748027393E+01
              7.5516402653E+00  3.0013700719E+00  8.9766200084E+00
              6.4090355722E+00  7.4843741588E+00  4.9295671605E+00
              4.3705611827E-01  7.6893073781E+00  1.2001555962E+01
              8.4238741309E+00  1.2232714786E+01  7.6995337657E+00
              5.8387974184E+00  5.9155119378E+00  1.4039991791E+01
              1.3107235988E+01  9.8055489044E+00  6.4400593019E-01
              8.3270647814E-01  1.7227458132E+01  1.2775664290E+01
              1.4372625432E+01  4.2560000137E+00  1.9730406948E+00
              5.7914453145E+00  4.0664955533E+00  3.9036518542E-01
              9.7815513593E+00  1.0257448955E+01  1.3164763822E+01
              1.6979663973E+01  2.5757556368E+00  1.2070399003E+01
              9.1476280310E-01  8.1192625454E+00  6.2498371664E+00
              8.8902943261E+00  1.3433615492E+01  1.7894990037E+01
              4.7238007437E+00  1.7074503731E+01  8.1487422033E+00
              1.1337419675E+00  1.7170180156E+01  3.2442179093E+00

# Energy cutoff for the planewaves

  ecut        32.0

# Parameters for the SCF cycles

  nstep       5
  diemac      4.0
  ixc         11






Here is the end of the message log for the 72 atoms cell with 4480 bands run 
on
16 processors :

================================================================================

 ----iterations are completed or convergence reached----

 outwf  : write wavefunction to file asio2test_para16_4096o_DS2_WFK
-P-0000  leave_test : synchronization done...
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 15 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
--------------------------------------------------------------------------
mpirun has exited due to process rank 1 with PID 6159 on
node node054 exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------
[green:29817] 15 more processes have sent help message help-mpi-api.txt /
mpi-abort
[green:29817] Set MCA parameter "orte_base_help_aggregate" to 0 to see all 
help
/ error messages





Here is the end of the log file I got from the 108 atoms cell (with 3200 
bands)
run on 32 processors :

================================================================================

 ----iterations are completed or convergence reached----

 outwf  : write wavefunction to file as108nr_004o_DS2_WFK
-P-0000  leave_test : synchronization done...
[node090:26797] *** Process received signal ***
[node090:26797] Signal: Segmentation fault (11)
[node090:26797] Signal code: Address not mapped (1)
[node090:26797] Failing at address: 0x2
[node090:26797] [ 0] /lib64/libpthread.so.0 [0x395300e4c0]
[node090:26797] [ 1]
/cvos/shared/apps/openmpi/intel/64/1.3.1/lib64/libmpi.so.0(ompi_ddt_add+0x6c1)
[0x2b389ea74951]
[node090:26797] [ 2]
/cvos/shared/apps/openmpi/intel/64/1.3.1/lib64/libmpi.so.0(ompi_ddt_create_indexed_block+0x1b3)
[0x2b389ea750c3]
[node090:26797] [ 3]
/cvos/shared/apps/openmpi/intel/64/1.3.1/lib64/libmpi.so.0(MPI_Type_create_indexed_block+0xb8)
[0x2b389ea9d848]
[node090:26797] [ 4]
/cvos/shared/apps/openmpi/intel/64/1.3.1/lib64/libmpi_f77.so.0(mpi_type_create_indexed_block_f+0x38)
[0x2b389e828780]
[node090:26797] [ 5]
/home/pcpm/waroquiers/583/abinit/5.8/bin/abinip(wffwritecg_+0xc29) [0x10823a9]
[node090:26797] [ 6]
/home/pcpm/waroquiers/583/abinit/5.8/bin/abinip(writewf_+0x1ef9) [0x107f0b7]
[node090:26797] [ 7]
/home/pcpm/waroquiers/583/abinit/5.8/bin/abinip(rwwf_+0x3e88) [0x107d1b4]
[node090:26797] [ 8]
/home/pcpm/waroquiers/583/abinit/5.8/bin/abinip(outwf_+0x2616) [0x5f5a72]
[node090:26797] [ 9]
/home/pcpm/waroquiers/583/abinit/5.8/bin/abinip(gstate_+0x15074) [0x4675cc]
[node090:26797] [10]
/home/pcpm/waroquiers/583/abinit/5.8/bin/abinip(driver_+0x740e) [0x44ea52]
[node090:26797] [11]
/home/pcpm/waroquiers/583/abinit/5.8/bin/abinip(MAIN__+0x52a6) [0x4448f6]
[node090:26797] [12] 
/home/pcpm/waroquiers/583/abinit/5.8/bin/abinip(main+0x2a)
[0x43f642]
[node090:26797] [13] /lib64/libc.so.6(__libc_start_main+0xf4) [0x395281d974]
[node090:26797] [14] /home/pcpm/waroquiers/583/abinit/5.8/bin/abinip 
[0x43f569]
[node090:26797] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 8 with PID 26797 on node node090 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------

<<winmail.dat>>